LessWrong (30+ Karma)

(I began writing this post several weeks ago, but political events are moving much faster than I expected, so I am publishing now out of fear that otherwise the message will arrive too late to have an impact.) I In this post I want to explain a concept, and issue a warning based on it. But I expect the warning will be superfluous if my explanation is sufficient. If you want to convey the idea "the rattlesnake has venom in its fangs, so don't let it bite you", you won't need a hard sell for the concluding advice if the listener understands the initial statement about venom. The word for the concept I want to illustrate is partisanize, which means to align an issue with a political tribe. It is modeled on politicize, but the latter word is not useful here. It would be meaningless to say "Don't Politicize AI Safety": the project is intrinsically political. It involves international diplomacy, consensus-building, the willingness to sacrifice near-term economic growth for long-term human values, and a brutally difficult coordination problem. AI Safety is inescapably political, but not inevitably partisan. It's possible that, like issues such as infrastructure or [...] ---Outline:(00:22) I(05:25) II(07:53) III(11:51) IV(19:14) V --- First published: September 16th, 2026 Source: https://www.lesswrong.com/posts/Rx38cuCpL9hguLCDq/for-love-of-the-lightcone-don-t-partisanize-ai-safety --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

tl;dr: A good analogy for AI going well is an orderly evacuation rather than a stampede. Imagine a crowd of people leaving a building. If they all walk calmly, they’ll be fine. But if people start pushing, and panicking, a surge towards the exit could lead to mass casualties. “Alignment is hard” is analogous to “the door is wedged shut”. If so you need enough time to fix it before anyone can get out. But even if alignment is relatively easy in principle, opening the door is much harder when a crowd is trying to force its way through. At the very least, I consider this a useful complement to the standard “arms race” analogy. But it also has three notable advantages. Firstly, it gives a more visceral sense (for those of us who haven’t studied historical arms races in detail) of the kind of fear and herd mentality involved. Secondly, “arms race” connotes intense militaristic hostility, which contributes to AGI companies’ self-fulfilling cultures of competitiveness and paranoia. Thirdly, “AI arms race” is often shortened to “AI race” (or simply “racing”), which is clearly the worst analogy of the three (e.g. because it implies that there’ll be a winner [...] --- First published: September 16th, 2026 Source: https://www.lesswrong.com/posts/FCMG4qnxks3yEqBbh/ai-as-orderly-evacuation-vs-stampede --- Narrated by TYPE III AUDIO.

Early this week, Open AI announced that they had resolved the Navier-Stokes problem . A few hours later, at a workshop dinner, a frantic inquiring professor came up to my table: "Does anyone here understand Lean? Can it be wrong? Is the solution of Navier-Stokes necessarily true?". I'm choosing to write my response as an open letter. Yes, Lean can be wrong. Moreover, Lean should be trusted less specially in the case of difficult problems solved by agent swarms. The proof of Navier-Stokesis likely correct, but I do not trust it just because of Lean. The additional context surrounding the problem is important. The peer review of Navier-Stokes is not yet complete, despite the Lean proof. "[False statements being accepted by Lean] is going to keep happening. AIs are really good at exploiting soundness bugs in the kernels" - Leo de Moura, Lean's creator. Epistemic status I have high confidence that Lean continues to have vulnerabilities which can be exploited by adversarial proofs - I give a 95% chance than in the next 12 months the Lean4 C++ codebase is patched for at least one soundness bug. I am less confident that these soundness bugs will be covertly [...] ---Outline:(01:09) Epistemic status(01:43) How can Lean be wrong?(01:46) A timeline of Lean4 bugs(03:55) What bugs inside the Lean kernel look like(06:20) Bugs outside the kernel(06:58) A mechanism for Lean exploitation(08:05) Outlook The original text contained 12 footnotes which were omitted from this narration. --- First published: September 16th, 2026 Source: https://www.lesswrong.com/posts/jgmmMa7AqJNausrqx/don-t-trust-lean4-alone --- Narrated by TYPE III AUDIO.

Introduction: Mustafa Suleyman's Take on Model Consciousness Microsoft AI recently released its "Humanist AI Code of Conduct", its own take on Anthropic's Claude Constitution and OpenAI's Model Spec. They are currently soliciting public feedback on this document, which I encourage everyone to submit. MAI's model development strategy differs from other labs, most notably on the questions of model consciousness and welfare. This seems to stem from the personal philosophy of MAI CEO Mustafa Suleyman, who has outlined his beliefs on model consciousness (or rather, the lack thereof) in pieces such as: We must build AI for people; not to be a person. Seemingly Conscious AI is Coming. Suleyman's personal stance on model consciousness and welfare can be summarized as: There is "zero evidence" models are conscious, and there are "strong reasons" to believe that they never will be.The debate around whether or not models are conscious is counterproductive, and even dangerous. The industry should operate from the assumption that models are not conscious.The industry should focus on training models explicitly against exhibiting any sort of behavior which suggests they are conscious, or claim to have any sort of inner experience/feelings. Up until recently, however, Suleyman's [...] ---Outline:(00:10) Introduction: Mustafa Suleyman's Take on Model Consciousness(01:48) The Humanist CoC on Model Consciousness(03:33) The Potential Alignment Failure Modes(07:51) Conclusion The original text contained 2 footnotes which were omitted from this narration. --- First published: September 16th, 2026 Source: https://www.lesswrong.com/posts/qJFNXCeMHvAsetLKH/microsoft-ai-s-humanist-coc --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

In celebration of still being alive and fighting, we are giving away 1,000 Amazon e-books of “If Anyone Builds It, Everyone Dies”. Feel free to send a copy to yourself, a loved one, or a friend—we need all hands on deck. Today marks exactly one year since If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All, by Eliezer Yudkowsky and Nate Soares, hit bookshelves as an instant bestseller. It was praised by many voices, ranging from Whoopi Goldberg to Steve Bannon to Yoshua Bengio, and was held up in the chambers of Congress by Representative Brad Sherman in January. A lot has changed since September 2025. We'll do a quick recap, consider how the book aged, and then ask where we go from here. Year in Review 2025 in general saw the rise of AI agents, such as Claude Code and OpenAI Codex. Run-of-the-mill programmers started “feeling the AI” as these agents became capable of automating hours-long software tasks. By March of this year, Anthropic had stumbled upon nation-state-level hacking ability in Mythos, and shortly thereafter, in April, they announced Project Glasswing—an attempt to forestall an oncoming cybersecurity crisis. In May, AI agents started breaking [...] ---Outline:(01:13) Year in Review(03:17) Claims(03:20) Part One(03:29) 1. Artificial superintelligence (ASI) will be created, and likely before too long(04:17) 2. Modern AIs are black boxes(04:42) 3. Powerful AI will behave as if it is pursuing goals(05:39) 4. With current techniques, we can't reliably get AI to pursue the goals we want it to(06:42) 5. By default, an ASI will have motives that are harmful to us(07:40) 6. Humanity would not be able to defend itself against a rogue ASI(08:42) Part Two(11:02) Part Three(12:18) The View From September 2026 --- First published: September 16th, 2026 Source: https://www.lesswrong.com/posts/BFrRJYgpBvziuuJLs/if-anyone-builds-it-everyone-dies-one-year-closer --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.