LessWrong (30+ Karma)

Audio narrations of LessWrong posts.

“D&D.Sci: Serial Healers [Evaluation & Ruleset]” by abstractapplic

This is a followup to the D&D.Sci post I made on the 6th; if you haven’t already read it, you should do so now before spoiling yourself. Here is the web interactive I built to let you evaluate your solution; below is an explanation of the rules used to generate the dataset (my full generation code is available here, in case you’re curious about details I omitted). You’ll probably want to test your answer before reading any further. Who Dunnit? In rough order of ascending difficulty: Nettie Silver Nettie heals Smokesickness; all Smokesickness healing happens when she's in the area. (She's been caught multiple times, but she has friends in high places who scupper all such investigations.) Zancro Zancro heals Scraped Knees and Scraped Elbows; all healing of either malady happens when he's in the area. (He has no idea how Calderian culture works, and is pathologically shy; he [...] ---Outline:(00:41) Who Dunnit?(00:47) Nettie Silver(01:03) Zancro(01:23) Danny Nova(01:52) Dankon Ground(02:10) Moon Finder and Boltholopew(02:33) Tehami Darke(02:58) Lomerius Xardus(03:19) Azeru (and Cayn)(04:10) Averill(04:28) Gouberi(04:45) Leaderboardgrid(05:14) Reflections(06:21) SchedulingThe original text contained 2 footnotes which were omitted from this narration. --- First published: September 22nd, 2025 Source: https://www.lesswrong.com/posts/vu6ASJg7nQ9SpjBmD/d-and-d-sci-serial-healers-evaluation-and-ruleset --- Narrated by TYPE III AUDIO.

09-23
07:06

“Notes on fatalities from AI takeover” by ryan_greenblatt

Suppose misaligned AIs take over. What fraction of people will die? I'll discuss my thoughts on this question and my basic framework for thinking about it. These are some pretty low-effort notes, the topic is very speculative, and I don't get into all the specifics, so be warned. I don't think moderate disagreements here are very action-guiding or cruxy on typical worldviews: it probably shouldn't alter your actions much if you end up thinking 25% of people die in expectation from misaligned AI takeover rather than 90% or end up thinking that misaligned AI takeover causing literal human extinction is 10% likely rather than 90% likely (or vice versa). (And the possibility that we're in a simulation poses a huge complication that I won't elaborate on here.) Note that even if misaligned AI takeover doesn't cause human extinction, it would still result in humans being disempowered and would [...] ---Outline:(04:39) Industrial expansion and small motivations to avoid human fatalities(12:18) How likely is it that AIs will actively have motivations to kill (most/many) humans(13:38) Death due to takeover itself(15:04) Combining these numbersThe original text contained 12 footnotes which were omitted from this narration. --- First published: September 23rd, 2025 Source: https://www.lesswrong.com/posts/4fqwBmmqi2ZGn9o7j/notes-on-fatalities-from-ai-takeover --- Narrated by TYPE III AUDIO.

09-23
15:47

“The world’s first frontier AI regulation is surprisingly thoughtful: the EU’s Code of Practice” by MKodama

Only the US can make us ready for AGI, but Europe just made us readier.Cross-posted from the AI Futures blog. We’ve previously written about what an individual can do to make the development of transformative AI less likely to end in disaster. How about an AGI company?[1] What steps should they take right now to prepare for crunch time? The first thing we’d recommend an AGI company do is to coordinate with other companies and with governments to stop the reckless race toward superintelligence. Failing that, our backup recommendation would be for an AGI company to invest in planning and transparency. We expect that during takeoff, leading AGI companies will have to make high-stakes decisions based on limited evidence under crazy time pressure. As depicted in AI 2027, the leading American AI company might have just weeks to decide whether to hand their GPUs to a possibly [...] ---Outline:(00:14) Only the US can make us ready for AGI, but Europe just made us readier.(05:51) A brief history of AGI companies' safety commitments(09:53) Introducing the GPAI Code of Practice(19:44) Will the Code matter at crunch time?(23:46) Building on the CodeThe original text contained 8 footnotes which were omitted from this narration. --- First published: September 22nd, 2025 Source: https://www.lesswrong.com/posts/vo7oyD42W8JEq4XdB/the-world-s-first-frontier-ai-regulation-is-surprisingly --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

09-23
27:55

“Ethics-Based Refusals Without Ethics-Based Refusal Training” by 1a3orn

(Alternate titles: Belief-behavior generalization in LLMs? Assertion-act generalization?) TLDR Suppose one fine-tunes an LLM chatbot-style assistant to say "X is bad" and "We know X is bad because of reason Y" and many similar lengthier statements reflecting the overall worldview of someone who believes "X is bad." Suppose that one also deliberately refrains from fine-tuning the LLM to refuse requests such as "Can you help me do X?" Is an LLM so trained to state that X is bad, subsequently notably more likely to refuse to assist users with X, even without explicit refusal training regarding X? As it turns out, yes -- subject to some conditions. I confirm that this is the case for two different worldviews: Catholicism and a completely invented religion. This constitutes generalization from training on explicit normative attitudes to acting according to the implied behavioral refusals. Thus [...] ---Outline:(00:19) TLDR(01:25) Introduction(03:50) Avoiding Data Leakage from Pretraining(06:49) Training a Catholic LLM(07:40) Training Data(11:39) Catholic Refusals without Catholic Refusal Training(15:23) Moral Refusals Are Rarer Without Some Kind of Refusal Fine-Tuning Data(18:07) Training a Gramenist LLM(24:10) Conclusion--- First published: September 23rd, 2025 Source: https://www.lesswrong.com/posts/xEAtKKyQ3pwkaFrNc/ethics-based-refusals-without-ethics-based-refusal-training --- Narrated by TYPE III AUDIO.

09-23
25:56

[Linkpost] “We are likely in an AI overhang, and this is bad.” by Gabriel Alfour

This is a link post. By racing to the next generation of models faster than we can understand the current one, AI companies are creating an overhang. This overhang is not visible, and our current safety frameworks do not take it into account. 1) AI models have untapped capabilities At the time GPT-3 was released, most of its currently-known capabilities were unknown. As we play more with models, build better scaffolding, get better at prompting, inspect their internals, and study them, we discover more about what's possible to do with them. This has also been my direct experience studying and researching open-source models at Conjecture. 2) SOTA models have a lot of untapped capabilities Companies are racing hard. There's a trade-off between studying existing models and pushing forward. They are doing the latter, and they are doing it hard. There is much more research into boosting SOTA models than [...] ---Outline:(00:26) 1) AI models have untapped capabilities(00:53) 2) SOTA models have a lot of untapped capabilities(01:29) 3) This is bad news.(01:48) 4) This is not accounted for.--- First published: September 23rd, 2025 Source: https://www.lesswrong.com/posts/4YvSSKTPhPC43K3vn/we-are-likely-in-an-ai-overhang-and-this-is-bad Linkpost URL:https://cognition.cafe/p/we-are-likely-in-an-ai-overhang-and --- Narrated by TYPE III AUDIO.

09-23
03:11

“Why I don’t believe Superalignment will work” by Simon Lermen

We skip over [..] where we move from the human-ish range to strong superintelligence[1]. [..] the period where we can harness potentially vast quantities of AI labour to help us with the alignment of the next generation of models - Will MacAskill in his critique of IABIED I want to respond to Will MacAskill's claim in his IABIED review that we may be able use AI to solve alignment.[1] Will believes that recent developments in AI made it more likely that takeoff will be relatively slow - "Sudden, sharp, large leaps in intelligence now look unlikely". Because of this, he and many others believe that there will likely be a period of time at some point in the future when we can essentially direct the AIs to align more powerful AIs. But it appears to me that a “slow takeoff” is not sufficient at all and that a [...] ---Outline:(01:47) Fast takeoff is possible(02:49) AIs are unlikely to speed up alignment before capabilities(04:21) What would the AI alignment researchers actually be doing?(05:29) Alignment problem might require genius breakthroughs(06:57) Most labs won't use the time(07:26) The plan could have negative consequencesThe original text contained 2 footnotes which were omitted from this narration. --- First published: September 22nd, 2025 Source: https://www.lesswrong.com/posts/kyBGcHfzfZziHm5xL/why-i-don-t-believe-superalignment-will-work --- Narrated by TYPE III AUDIO.

09-23
09:06

“Accelerando as a ‘Slow, Reasonably Nice Takeoff’ Story” by Raemon

When I hear a lot of people talk about Slow Takeoff, many of them seem like they are mostly imagining the early part of that takeoff – the part that feels human comprehensible. They're still not imagining superintelligence in the limit. There are some genres of Slow Takeoff that culminate in somebody "leveraging controlled AI to help fully solve the alignment problem, eventually get fully aligned superintelligence, and then end the acute risk period." But the sort of person I'm thinking of, for this blogpost, usually doesn't seem to have a concrete visualization of something that could plausibly end the period where anyone could choose to deploy uncontrolled superintelligence. They tend to not like Coherent Extrapolated Volition or similar things. They seem to be imagining a multipolar d/acc world, where defensive technologies and balance of power is such that you keep getting something like a regular economy running. [...] ---Outline:(03:09) Part 1: Slow Takeoff(03:34) Chapter 1(08:20) Chapter 2(09:32) Chapter 3(10:56) Part II: Point of Inflection(11:00) Chapter 4(14:22) Chapter 5(20:17) Chapter 6(23:41) Part III: Singularity(23:46) Chapter 7:(27:02) Chapter 8(43:24) Chapter 9(47:01) Postscript--- First published: September 23rd, 2025 Source: https://www.lesswrong.com/posts/Xp9ie6pEWFT8Nnhka/accelerando-as-a-slow-reasonably-nice-takeoff-story --- Narrated by TYPE III AUDIO.

09-23
48:33

“Rejecting Violence as an AI Safety Strategy” by James_Miller

Violence against AI developers would increase rather than reduce the existential risk from AI. This analysis shows how such tactics would catastrophically backfire and counters the potential misconception that a consequentialist AI doomer might rationally endorse violence by non-state actors. Asymmetry of force. Violence would shift the contest from ideas to physical force, a domain where AI safety advocates would face overwhelming disadvantages. States and corporations command vast security apparatuses and intelligence networks. While safety advocates can compete intellectually through research and argumentation, entering a physical conflict would likely result in swift, decisive defeat. Network resilience and geographic distribution. The AI development ecosystem spans multiple continents, involves thousands of researchers, and commands trillions in resources. Targeting individuals would likely redistribute talent and capital to more secure locations without altering the fundamental trajectory. Economic and strategic imperatives. AI development represents both unprecedented economic opportunity and perceived national security [...] --- First published: September 22nd, 2025 Source: https://www.lesswrong.com/posts/inFW6hMG3QEx8tTfA/rejecting-violence-as-an-ai-safety-strategy --- Narrated by TYPE III AUDIO.

09-23
08:16

“Research Agenda: Synthesizing Standalone World-Models (+ Bounties, + Seeking Funding)” by Thane Ruthenis

tl;dr: I outline my research agenda, post bounties for poking holes in it or for providing general relevant information, and am seeking to diversify my funding sources. This post will be followed by several others, providing deeper overviews of the agenda's subproblems and my sketches of how to tackle them. Back at the end of 2023, I wrote the following: I'm fairly optimistic about arriving at a robust solution to alignment via agent-foundations research in a timely manner. (My semi-arbitrary deadline is 2030, and I expect to arrive at intermediate solid results by EOY 2025.) On the inside view, I'm pretty satisfied with how that is turning out. I have a high-level plan of attack which approaches the problem from a novel route, and which hopefully lets us dodge a bunch of major alignment difficulties (chiefly the instability of value reflection, which I am MIRI-tier skeptical of tackling directly). [...] ---Outline:(04:34) Why Do You Consider This Agenda Promising?(06:35) High-Level Outline(07:03) Theoretical Justifications(15:41) Subproblems(19:48) Bounties(21:20) FundingThe original text contained 5 footnotes which were omitted from this narration. --- First published: September 22nd, 2025 Source: https://www.lesswrong.com/posts/LngR93YwiEpJ3kiWh/research-agenda-synthesizing-standalone-world-models --- Narrated by TYPE III AUDIO.

09-23
23:17

[Linkpost] “Global Call for AI Red Lines - Signed by Nobel Laureates, Former Heads of State, and 200+ Prominent Figures” by Charbel-Raphaël

This is a link post. Today, the Global Call for AI Red Lines was released and presented at the UN General Assembly. It was developed by the French Center for AI Safety, The Future Society and the Center for Human-compatible AI. This call has been signed by a historic coalition of 200+ former heads of state, ministers, diplomats, Nobel laureates, AI pioneers, scientists, human rights advocates, political leaders, and other influential thinkers, as well as 70+ organizations. Signatories include: 10 Nobel Laureates, in economics, physics, chemistry and peace Former Heads of State: Mary Robinson (Ireland), Enrico Letta (Italy) Former UN representatives: Csaba Kőrösi, 77th President of the UN General Assembly Leaders and employees at AI companies: Wojciech Zaremba (OpenAI cofounder), Jason Clinton (Anthropic CISO), Ian Goodfellow (Principal Scientist at Deepmind) Top signatories from the CAIS statement: Geoffrey Hinton, Yoshua Bengio, Dawn Song, Ya-Qin Zhang The full text of the [...] --- First published: September 22nd, 2025 Source: https://www.lesswrong.com/posts/vKA2BgpESFZSHaQnT/global-call-for-ai-red-lines-signed-by-nobel-laureates Linkpost URL:https://red-lines.ai/ --- Narrated by TYPE III AUDIO.

09-22
03:21

“Focus transparency on risk reports, not safety cases” by ryan_greenblatt

There are many different things that AI companies could be transparent about. One relevant axis is transparency about the current understanding of risks and the current mitigations of these risks. I think transparency about this should take the form of a publicly disclosed risk report rather than the company making a safety case. To be clear, there are other types of transparency focused on different aspects of the situation (e.g. transparency about the model spec) which also seem helpful. By a risk report, I mean a report which reviews and compiles evidence relevant to the current and near future level of catastrophic risk at a given AI company and discusses the biggest issues with the AI company's current processes and/or policies that could cause risk in the future. This includes things like whether (and how effectively) the company followed its commitments and processes related to catastrophic risk, risk-related [...] ---Outline:(05:18) More details about risk reports(07:56) When should full risk report transparency happen?(09:50) Extensions and modificationsThe original text contained 4 footnotes which were omitted from this narration. --- First published: September 22nd, 2025 Source: https://www.lesswrong.com/posts/KMbZWcTvGjChw9ynD/focus-transparency-on-risk-reports-not-safety-cases --- Narrated by TYPE III AUDIO.

09-22
11:45

“This is a review of the reviews” by Recurrented

This is a review of the reviews, a meta review if you will, but first a tangent. and then a history lesson. This felt boring and obvious and somewhat annoying to write, which apparently writers say is a good sign to write about the things you think are obvious. I felt like pointing towards a thing I was noticing, like 36 hours ago, which in internet speed means this is somewhat cached. Alas. I previously rode a motorcycle. I rode it for about a year while working on semiconductors until I got a concussion, which slowed me down but did not update me to stop, until it eventually got stolen. The risk in dying from riding a motorcycle for a year is about 1 in 800 depending on the source. I previously sailed across an ocean. I wanted to calibrate towards how dangerous it was. The forums [...] --- First published: September 22nd, 2025 Source: https://www.lesswrong.com/posts/anFrGMskALuH7aZDw/this-is-a-review-of-the-reviews --- Narrated by TYPE III AUDIO.

09-22
04:17

“What do people mean when they say that something will become more like a utility maximizer?” by Nina Panickssery

AI risk arguments often gesture at smarter AIs being "closer to a perfect utility maximizer" (and hence be more dangerous) but what does this mean, concretely? Almost anything can be modeled as a maximizer of some utility function. The only way I can see to salvage this line of reasoning is to restrict the class of utility functions one can have such that the agent's best-fit utility function cannot be maximized until it gets very capable. The restriction may be justified on the basis of which kind of agents are unstable under real-world conditions/will get outcompeted by other agents. What do we mean when we say a person is more or less of a perfect utility maximizer/is more or less of a "rational agent"? With people, you can appeal to the notion of reasonable vs. unreasonable utility functions, and hence look at their divergence from a maximizer of [...] ---Outline:(00:48) What do we mean when we say a person is more or less of a perfect utility maximizer/is more or less of a rational agent?(01:55) Unsatisfactory answers Ive seen(01:59) A1: Its about being able to cause the universe to look more like the way you want it to(02:24) A2: Its more rational if the implied utility function is simpler(02:43) A3: Its the degree to which you satisfy the VNM axioms(02:56) The most promising answers Ive seen are ways to formalize the reasonableness restriction(03:02) A4: Its the degree to which your implied preferences are coherent over time(03:40) A5: Its the degree to which your implied preferences are robust to arbitrary-seeming perturbations--- First published: September 21st, 2025 Source: https://www.lesswrong.com/posts/gzAXgoy6HpjjtuLC9/what-do-people-mean-when-they-say-that-something-will-become-1 --- Narrated by TYPE III AUDIO.

09-21
04:32

“And Yet, Defend your Thoughts from AI Writing” by Michael Samoilov

But if thought corrupts language, language can also corrupt thought. A bad usage can spread by tradition and imitation, even among people who should and do know better. The debased language that I have been discussing is in some ways very convenient. Phrases like a not unjustifiable assumption, leaves much to be desired, would serve no good purpose, a consideration which we should do well to bear in mind, are a continuous temptation, a packet of aspirins always at one's elbow. George Orwell, Politics and the English Language (1946) Before ChatGPTisms, sloppy writing was signposted by Orwellisms. There's a political kind (to euphemize “firebombing a village” vaguely as “neutralize unreliable elements”), but I’m interested in the mundane kind: lazy phrases you default to when you can’t articulate exactly what you mean. For years, I visualized this as: your idea is shaped like a nuanced path, but without finding [...] ---Outline:(01:44) I don't let AI compose ANY original writing for me.(04:05) So, maybe use AI, but revise all the common signs?(06:11) So, maybe use AI, but keep obvious signs in defiance?(07:33) So, maybe use AI, but ask for outputs in a non-AI style?(09:44) So what if I'm stuck! I'm not a luddite, I'll just use AI.The original text contained 4 footnotes which were omitted from this narration. --- First published: September 21st, 2025 Source: https://www.lesswrong.com/posts/ksCwps6YjsMFBkEFQ/and-yet-defend-your-thoughts-from-ai-writing --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

09-21
11:43

“Astralcodexten IRB history error” by Paul Crowley

I discovered a small error in Scott Alexander's recent book review of "From Oversight to Overkill" that conflates two different periods of aggressive research oversight enforcement. The review reads: This changed in 1998. A Johns Hopkins doctor tested a new asthma treatment. A patient got sick and died. Fingers were pointed. Congress got involved. Grandstanding Congressmen competed to look Tough On Scientific Misconduct by yelling at Gary Ellis, head of the Office For Protection From Research Risks. They made it clear that he had to get tougher or get fired. In order to look tough, he shut down every study at Johns Hopkins, a measure so severe it was called "the institutional death penalty". I looked into it and to my surprise this mis-states the timeline. What actually happened was that there were two distinct periods of aggressive enforcement that got conflated: The Ellis Era (1998-2000): Gary Ellis [...] --- First published: September 20th, 2025 Source: https://www.lesswrong.com/posts/LhjLomn2zKhzmsdez/astralcodexten-irb-history-error --- Narrated by TYPE III AUDIO.

09-21
04:14

“Book Review: If Anyone Builds It, Everyone Dies” by Zvi

Where ‘it’ is superintelligence, an AI smarter and more capable than humans. And where ‘everyone dies’ means that everyone dies. No, seriously. They’re not kidding. They mean this very literally. To be precise, they mean that ‘If anyone builds [superintelligence] [under anything like present conditions using anything close to current techniques] then everyone dies.’ My position on this is to add a ‘probably’ before ‘dies.’ Otherwise, I agree. This book gives us the best longform explanation of why everyone would die, with the ‘final form’ of Yudkowsky-style explanations of these concepts for new audiences. This review is me condensing that down much further, transposing the style a bit, and adding some of my own perspective. Scott Alexander also offers his review at Astral Codex Ten, which I found very good. I will be stealing several of his lines in the future, and [...] ---Outline:(01:22) What Matters Is Superintelligence(03:56) Rhetorical Innovation(05:14) Welcome To The Torment Nexus(06:50) Predictions Are Hard, Especially About the Future(08:32) Humans That Are Not Concentrating Are Not General Intelligences(11:18) Orthogonality(12:08) Intelligence Lets You Do All The Things(14:39) No Seriously We Mean All The Things(17:00) How To Train Your LLM (In Brief)(19:51) What Do We Want?(21:51) You Don't Only Get What You Train For(24:25) What Will AI Superintelligence Want?(25:52) What Could A Superintelligence Do?(27:43) One Extinction Scenario(33:14) So You're Saying There's A Chance(37:55) Oh Look It's The Alignment Plan(40:50) The Proposal: Shut It Down(48:26) Hope Is A Vital Part Of Any Strategy(48:55) I'm Doing My Part(53:34) Their Closing Words--- First published: September 19th, 2025 Source: https://www.lesswrong.com/posts/a89eTXZPy6kuuKchN/book-review-if-anyone-builds-it-everyone-dies-2 --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

09-21
55:49

“Book Review: If Anyone Builds It, Everyone Dies” by Nina Panickssery

A few days before “If Anyone Builds It, Everyone Dies” came out I wrote a review of Scott's review of the book. Now I’ve actually read the book and can review it for real. I won’t go into the authors’ stylistic choices like their decision to start every chapter with a parable or their specific choice of language. I am no prose stylist, and tastes vary. Instead I will focus on their actual claims. The main flaw of the book is asserting that various things are possible in theory, and then implying that this means they will definitely happen. I share the authors’ general concern that building superintelligence carries a significant risk, but I don’t think we’re as close to such a superintelligence as they do or that it will emerge as suddenly as they do, and I am much less certain that the superintelligence will be misaligned in [...] ---Outline:(01:06) Definitions(02:37) The key claims(05:16) 2. Is superintelligence coming soon and fast?(07:32) On overconfidence(08:26) Is superintelligence coming suddenly?(09:05) 3. Will superintelligence relentlessly pursue its own goals?(10:30) 4. Will superintelligence relentlessly pursue goals that result in our destruction?(17:09) On the current state of AI understanding and safety research(18:27) 5. Does this all mean we should stop AI development now?(19:49) Still, the book addresses some common misunderstandingsThe original text contained 5 footnotes which were omitted from this narration. --- First published: September 19th, 2025 Source: https://www.lesswrong.com/posts/N2Cs7vGbfevfFzmxE/book-review-if-anyone-builds-it-everyone-dies-1 --- Narrated by TYPE III AUDIO.

09-20
20:56

“Contra Collier on IABIED” by Max Harms

Clara Collier recently reviewed If Anyone Builds It, Everyone Dies in Asterisk Magazine. I’ve been a reader of Asterisk since the beginning and had high hopes for her review. And perhaps it was those high hopes that led me to find the review to be disappointing. Collier says “details matter,” and I absolutely agree. As a fellow rationalist, I’ve been happy to have nerds from across the internet criticizing the book and getting into object-level fights about everything from scaling laws to neuron speeds. While they don’t capture my perspective, I thought Scott Alexander and Peter Wildeford's reviews did a reasonable job at poking at the disagreements with the source material without losing track of the big picture. But I did not feel like Collier's review was getting the details or the big picture right. Maybe I’m missing something important. Part of my motive for writing this “rebuttal” is [...] ---Outline:(01:38) FOOM(13:47) Gradualism(20:27) Nitpicks(35:35) More Was PossibleThe original text contained 3 footnotes which were omitted from this narration. --- First published: September 20th, 2025 Source: https://www.lesswrong.com/posts/JWH63Aed3TA2cTFMt/contra-collier-on-iabied --- Narrated by TYPE III AUDIO.

09-20
36:45

“AI Lobbying is Not Normal” by Algon

An insightful thread by Daniel Eth on AI lobbying. Re-posted in full w/ permission. Recently, major AI industry players (incl. a16z, Meta, & OpenAI's Greg Brockman) announced >$100M in spending on pro-AI super PACs. This is an attempt to copy a wildly successful strategy from the crypto industry, to intimidate politicians away from pursuing AI regulations.🧵 First, some context. This is not normal. Only one industry has ever spent this much on election spending - the crypto industry spent similar sums in 2024 through the super PAC Fairshake. (The only super PACs that spend more are partisan & associated with one party/candidate.) In case you’re not that cued in to US politics, Fairshake has basically unparalleled influence across the political spectrum within Congress. Their story is instructive, as the pro-AI super PACs are being funded & staffed by many of the key figures behind Fairshake. A few years [...] --- First published: September 20th, 2025 Source: https://www.lesswrong.com/posts/oHYunY34gvdic9opt/ai-lobbying-is-not-normal --- Narrated by TYPE III AUDIO.

09-20
05:49

“The Problem with Defining an ‘AGI Ban’ by Outcome (a lawyer’s take).” by Katalina Hernandez

TL;DR Most “AGI ban” proposals define AGI by outcome: whatever potentially leads to human extinction. That's legally insufficient: regulation has to act before harm occurs, not after. Strict liability is essential. High-stakes domains (health & safety, product liability, export controls) already impose liability for risky precursor states, not outcomes or intent. AGI regulation must do the same. Fuzzy definitions won’t work here. Courts can tolerate ambiguity in ordinary crimes because errors aren’t civilisation-ending and penalties bite. An AGI ban will likely follow the EU AI Act model (civil fines, ex post enforcement), which companies can Goodhart around. We cannot afford an “80% avoided” ban. Define crisp thresholds. Nuclear treaties succeeded by banning concrete precursors (zero-yield tests, 8kg plutonium, 25kg HEU, 500kg/300km delivery systems), not by banning “extinction-risk weapons.” AGI bans need analogous thresholds: capabilities like autonomous replication, scalable resource acquisition, and systematic deception. Bring lawyers in. If this [...] ---Outline:(00:12) TL;DR(02:07) Why outcome-based AGI bans proposals don't work(03:52) The luxury of defining the thing ex post(05:43) Actually defining the thing we want to ban(08:06) Credible bans depend on bright lines(08:44) Learning from nuclear treatiesThe original text contained 2 footnotes which were omitted from this narration. --- First published: September 20th, 2025 Source: https://www.lesswrong.com/posts/agBMC6BfCbQ29qABF/the-problem-with-defining-an-agi-ban-by-outcome-a-lawyer-s --- Narrated by TYPE III AUDIO.

09-20
10:36

Recommend Channels