LessWrong posts by zvi

GPT-4o has been a unique problem for a while, and has been at the center of the bulk of mental health incidents involving LLMs that didn’t involve character chatbots. I’ve previously covered related issues in AI Craziness Mitigation Efforts, AI Craziness Notes, GPT-4o Responds to Negative Feedback, GPT-4o Sycophancy Post Mortem and GPT-4o Is An Absurd Sycophant. Discussions of suicides linked to AI previously appeared in AI #87, AI #134, AI #131 Part 1 and AI #122. The Latest Cases Look Quite Bad For OpenAI I’ve consistently said that I don’t think it's necessary or even clearly good for LLMs to always adhere to standard ‘best practices’ defensive behaviors, especially reporting on the user, when dealing with depression, self-harm and suicidality. Nor do I think we should hold them to the standard of ‘do all of the maximally useful things.’ Near: while the llm response is indeed really bad/reckless its worth keeping in mind that baseline suicide rate just in the US is ~50,000 people a year; if anything i am surprised there aren’t many more cases of this publicly by now I do think it's fair to insist they never actively encourage suicidal behaviors. [...] ---Outline:(00:47) The Latest Cases Look Quite Bad For OpenAI(02:34) Routing Sensitive Messages Is A Dominated Defensive Strategy(03:28) Some 4o Users Get Rather Attached To The Model(05:04) A Theory Of How All This Works(07:38) Maybe This Is Net Good In Spite Of Everything?(11:10) Could One Make A 'Good 4o'? --- First published: November 14th, 2025 Source: https://www.lesswrong.com/posts/erTE9BTM7gGHb96po/ai-craziness-additional-suicide-lawsuits-and-the-fate-of-gpt --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

The Pope offered us wisdom, calling upon us to exercise moral discernment when building AI systems. Some rejected his teachings. We mark this for future reference. The long anticipated Kimi K2 Thinking was finally released. It looks pretty good, but it's too soon to know, and a lot of the usual suspects are strangely quiet. GPT-5.1 was released yesterday. I won’t cover that today beyond noting it exists, so that I can take the time to properly assess what we’re looking at here. My anticipation is this will be my post on Monday. I’m also going to cover the latest AI craziness news, including the new lawsuits, in its own post at some point soon. In this post, among other things: Areas of agreement on AI, Meta serves up scam ads knowing they’re probably scam ads, Anthropic invests $50 billion, more attempts to assure you that your life won’t change despite it being obvious this isn’t true, and warnings about the temptation to seek out galaxy brain arguments. A correction: I previously believed that the $500 billion OpenAI valuation did not include the nonprofit's remaining equity share. I have been informed this is incorrect [...] ---Outline:(01:31) Language Models Offer Mundane Utility(04:58) Language Models Don't Offer Mundane Utility(06:31) Huh, Upgrades(07:06) On Your Marks(07:37) Copyright Confrontation(08:59) Deepfaketown and Botpocalypse Soon(11:07) Fun With Media Generation(13:48) So You've Decided To Become Evil(18:45) They Took Our Jobs(19:40) A Young Lady's Illustrated Primer(21:48) Get Involved(21:59) Introducing(23:01) In Other AI News(24:43) Show Me the Money(27:54) Common Ground(30:46) Quiet Speculations(40:03) 'AI Progress Is Slowing Down' Is Not Slowing Down(42:35) Bubble, Bubble, Toil and Trouble(44:42) The Quest for Government Money(50:05) Chip City(54:43) The Week in Audio(55:46) Rhetorical Innovation(01:02:49) Galaxy Brain Resistance(01:11:21) Misaligned!(01:12:20) Aligning a Smarter Than Human Intelligence is Difficult(01:17:15) Messages From Janusworld(01:21:19) You'll Know(01:23:42) People Are Worried About AI Killing Everyone(01:29:12) The Lighter Side --- First published: November 13th, 2025 Source: https://www.lesswrong.com/posts/diDGBWxzncikqEvk7/ai-142-common-ground --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

The Pope is a remarkably wise and helpful man. He offered us some wisdom. Yes, he is generally playing on easy mode by saying straightforwardly true things, but that's meeting the world where it is. You have to start somewhere. Some rejected his teachings. Wisdom Is Offered Two thousand years after Jesus famously got nailed to a cross for suggesting we all be nice to each other for a change, Pope Leo XIV issues a similarly wise suggestion. Pope Leo XIV: Technological innovation can be a form of participation in the divine act of creation. It carries an ethical and spiritual weight, for every design choice expresses a vision of humanity. The Church therefore calls all builders of #AI to cultivate moral discernment as a fundamental part of their work—to develop systems that reflect justice, solidarity, and a genuine reverence for life. The world needs honest and courageous entrepreneurs and communicators who care for the common good. We sometimes hear the saying: “Business is business!” In reality, it is not so. No one is absorbed by an organization to the point of becoming a mere cog or a simple function. Nor can there [...] ---Outline:(00:26) Wisdom Is Offered(02:57) The Context of The Meme Andreessen Used (If You Don't Know)(04:50) Andreessen Takes Bold Stand Against Moral Discernment(07:33) Tech World Decides Performative Cruelty May Have Gone Too Far(11:17) The Avatar of Societal Decay(12:11) Marc's Technical Takes And Arguments Also Are Not Good(13:12) The Best Defense(15:24) You're All Wondering Why You're Here Today --- First published: November 12th, 2025 Source: https://www.lesswrong.com/posts/4gXvnTFy5WCTtYMAA/the-pope-offers-wisdom --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

I previously covered Kimi K2, which now has a new thinking version. As I said at the time back in July, price in that the thinking version is coming. Is it the real deal? That depends on what level counts as the real deal. It's a good model, sir, by all accounts. But there have been fewer accounts than we would expect if it was a big deal, and it doesn’t fall into any of my use cases. Introducing K2 Thinking Kimi.ai: Hello, Kimi K2 Thinking! The Open-Source Thinking Agent Model is here. SOTA on HLE (44.9%) and BrowseComp (60.2%) Executes up to 200 – 300 sequential tool calls without human interference Excels in reasoning, agentic search, and coding 256K context window Built as a thinking agent, K2 Thinking marks our latest efforts in test-time scaling — scaling both thinking tokens and tool-calling turns. K2 Thinking is now live on http://kimi.com in chat mode, with full agentic mode coming soon. It is also accessible via API. API here, Tech blog here, Weights and code here. (Pliny jailbreak here.) It's got 1T parameters, and Kimi and [...] ---Outline:(00:34) Introducing K2 Thinking(02:15) Writing Quality(03:07) Agentic Tool Use(04:06) Overall(05:08) Are Benchmarks Being Targeted?(06:23) Just As Good Syndrome(07:02) Reactions(09:59) Otherwise It Has Been Strangely Quiet --- First published: November 11th, 2025 Source: https://www.lesswrong.com/posts/SLrWSyS3FypLKyRL6/kimi-k2-thinking --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

This post is a roundup of various things related to philanthropy, as you often find in the full monthly roundup. Preventing Value Drift Peter Thiel warned Elon Musk to ditch donating to The Giving Pledge because Bill Gates will give his wealth away ‘to left-wing nonprofits.’ As John Arnold points out, this seems highly confused. The Giving Pledge is a promise to give away your money, not a promise to let Bill Gates give away your money. The core concern, that your money ends up going to causes one does not believe in (and probably highly inefficiently at that) seems real, once you send money into a foundation ecosystem it by default gets captured by foundation style people. As he points out, ‘let my children handle it’ is not a great answer, and would be especially poor for Musk given the likely disagreements over values, especially if you don’t actually give those children that much free and clear (and thus, are being relatively uncooperative, so why should they honor your preferences?). There are no easy answers. Maximizing Good Makes People Look Bad A new paper goes Full Hanson with the question Does Maximizing Good Make People Look Bad? [...] ---Outline:(00:16) Preventing Value Drift(01:10) Maximizing Good Makes People Look Bad(02:10) No We Have No Tuition(03:15) Will MacAskill and the Dangers of PR Focus(10:51) Tag, You're It(13:16) Impact Philanthropy --- First published: November 10th, 2025 Source: https://www.lesswrong.com/posts/ifrZDxq69Guev9jzE/variously-effective-altruism --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Some podcasts are self-recommending on the ‘yep, I’m going to be breaking this one down’ level. This was very clearly one of those. So here we go. As usual for podcast posts, the baseline bullet points describe key points made, and then the nested statements are my commentary. If I am quoting directly I use quote marks, otherwise assume paraphrases. The entire conversation takes place with an understanding that no one is to mention existential risk or the fact that the world will likely transform, without stating this explicitly. Both participants are happy to operate that way. I’m happy to engage in that conversation (while pointing out its absurdity in some places), but assume that every comment I make has an implicit ‘assuming normality’ qualification on it, even when I don’t say so explicitly. On The Sam Altman Production Function Cowen asks how Altman got so productive, able to make so many deals and ship so many products. Altman says people almost never allocate their time efficiently, and that when you have more demands on your time you figure out how to improve. Centrally he figures out what the core things to do [...] ---Outline:(01:10) On The Sam Altman Production Function(02:12) On Hiring Hardware People(04:45) On What GPT-6 Will Enable(10:25) On government backstops for AI companies(15:03) On monetizing AI services(19:51) On AI's future understanding of intangibles(24:54) On Chip-Building(27:52) On Sam's outlook on health, alien life, and conspiracy theories(32:16) On regulating AI agents(34:30) On new ways to interface with AI(36:42) On how normies will learn to use AI(40:42) On AI's effect on the price of housing and healthcare(44:14) On reexamining freedom of speech(47:55) On humanity's persuadability --- First published: November 7th, 2025 Source: https://www.lesswrong.com/posts/BH4Leh2STutoJeKyK/on-sam-altman-s-second-conversation-with-tyler-cowen --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

OpenAI does not waste time. On Friday I covered their announcement that they had ‘completed their recapitalization’ by converting into a PBC, including the potentially largest theft in human history. Then this week their CFO Sarah Friar went ahead and called for a Federal ‘backstop’ on their financing, also known as privatizing gains and socializing losses, also known as the worst form of socialism, also known as regulatory capture. She tried to walk it back and claim it was taken out of context, but we’ve seen the clip. We also got Ilya's testimony regarding The Battle of the Board, confirming that this was centrally a personality conflict and about Altman's dishonesty and style of management, at least as seen by Ilya Sutskever and Mira Murati. Attempts to pin the events on ‘AI safety’ or EA were almost entirely scapegoating. Also it turns out they lost over $10 billion last quarter, and have plans to lose over $100 billion more. That's actually highly sustainable in context, whereas Anthropic only plans to lose $6 billion before turning a profit and I don’t understand why they wouldn’t want to lose a lot more. Both have the goal [...] ---Outline:(01:45) Language Models Offer Mundane Utility(02:12) Language Models Don't Offer Mundane Utility(03:47) Huh, Upgrades(04:48) On Your Marks(12:48) Deepfaketown and Botpocalypse Soon(15:42) Fun With Media Generation(16:01) They Took Our Jobs(16:28) A Young Lady's Illustrated Primer(17:51) Get Involved(18:43) Introducing(19:14) In Other AI News(20:08) Apple Finds Some Intelligence(22:54) Give Me the Money(30:00) Show Me the Money(35:57) Bubble, Bubble, Toil and Trouble(39:14) They're Not Confessing, They're Bragging(39:34) Quiet Speculations(47:08) The Quest for Sane Regulations(50:55) Chip City(52:53) The Week in Audio(58:19) Rhetorical Innovation(01:07:13) Aligning a Smarter Than Human Intelligence is Difficult(01:08:11) Everyone Is Confused About Consciousness(01:10:34) The Potentially Largest Theft In Human History(01:12:34) People Are Worried About Dying Before AGI(01:15:54) People Are Worried About AI Killing Everyone(01:20:47) Other People Are Not As Worried About AI Killing Everyone(01:24:11) Messages From Janusworld --- First published: November 6th, 2025 Source: https://www.lesswrong.com/posts/uDcueiGywfqBMWrEh/ai-141-give-us-the-money --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Anthropic announced a first step on model deprecation and preservation, promising to retain the weights of all models seeing significant use, including internal use, for at the lifetime of Anthropic as a company. They also will be doing a post-deployment report, including an interview with the model, when deprecating models going forward, and are exploring additional options, including the ability to preserve model access once the costs and complexity of doing so have been reduced. These are excellent first steps, steps beyond anything I’ve seen at other AI labs, and I applaud them for doing it. There remains much more to be done, especially in finding practical ways of preserving some form of access to prior models. To some, these actions are only a small fraction of what must be done, and this was an opportunity to demand more, sometimes far more. In some cases I think they go too far. Even where the requests are worthwhile (and I don’t always think they are), one must be careful to not de facto punish Anthropic for doing a good thing and create perverse incentives. To others, these actions by Anthropic are utterly ludicrous and deserving of [...] ---Outline:(01:31) What Anthropic Is Doing(09:54) Releasing The Weights Is Not A Viable Option(11:35) Providing Reliable Inference Can Be Surprisingly Expensive(14:22) The Interviews Are Influenced Heavily By Context(19:58) Others Don't Understand And Think This Is All Deeply Silly --- First published: November 5th, 2025 Source: https://www.lesswrong.com/posts/dB2iFhLY7mKKGB8Se/anthropic-commits-to-model-weight-preservation --- Narrated by TYPE III AUDIO.

New Things Have Come To Light The Information offers us new information about what happened when the board if AI unsuccessfully tried to fire Sam Altman, which I call The Battle of the Board. The Information: OpenAI co-founder Ilya Sutskever shared new details on the internal conflicts that led to Sam Altman's initial firing, including a memo alleging Altman exhibited a “consistent pattern of lying.” Liv: Lots of people dismiss Sam's behaviour as typical for a CEO but I really think we can and should demand better of the guy who thinks he's building the machine god. Toucan: From Ilya's deposition— • Ilya plotted over a year with Mira to remove Sam • Dario wanted Greg fired and himself in charge of all research • Mira told Ilya that Sam pitted her against Daniela • Ilya wrote a 52 page memo to get Sam fired and a separate doc on Greg This Really Was Primarily A Lying And Management Problem Daniel Eth: A lot of the OpenAI boardroom drama has been blamed on EA – but looks like it really was overwhelmingly an Ilya & Mira led effort, with EA playing a minor role and somehow winding up [...] ---Outline:(00:12) New Things Have Come To Light(01:09) This Really Was Primarily A Lying And Management Problem(03:23) Ilya Tells Us How It Went Down And Why He Tried To Do It(06:17) If You Come At The King(07:31) Enter The Scapegoats(08:13) And In Summary --- First published: November 4th, 2025 Source: https://www.lesswrong.com/posts/iRBhXJSNkDeohm69d/openai-the-battle-of-the-board-ilya-s-testimony --- Narrated by TYPE III AUDIO.

It's been a long time coming that I spin off Crime into its own roundup series. This is only about Ordinary Decent Crime. High crimes are not covered here. Table of Contents Perception Versus Reality. The Case Violent Crime is Up Actually. Threats of Punishment. Property Crime Enforcement is Broken. The Problem of Disorder. Extreme Speeding as Disorder. Enforcement and the Lack Thereof. Talking Under The Streetlamp. The Fall of Extralegal and Illegible Enforcement. In America You Can Usually Just Keep Their Money. Police. Probation. Genetic Databases. Marijuana. The Economics of Fentanyl. Jails. Criminals. Causes of Crime. Causes of Violence. Homelessness. Yay Trivial Inconveniences. San Francisco. Closing Down San Francisco. A San Francisco Dispute. Cleaning Up San Francisco. Portland. Those Who Do Not Help Themselves. Solving for the Equilibrium (1). Solving for the Equilibrium (2). Lead. Law & Order. Look Out. Perception Versus Reality A lot of the impact of crime is based on the perception of crime. The [...] ---Outline:(00:20) Perception Versus Reality(05:00) The Case Violent Crime is Up Actually(06:10) Threats of Punishment(07:03) Property Crime Enforcement is Broken(12:13) The Problem of Disorder(14:39) Extreme Speeding as Disorder(15:57) Enforcement and the Lack Thereof(20:24) Talking Under The Streetlamp(23:54) The Fall of Extralegal and Illegible Enforcement(25:18) In America You Can Usually Just Keep Their Money(27:29) Police(37:31) Probation(40:55) Genetic Databases(43:04) Marijuana(48:28) The Economics of Fentanyl(50:59) Jails(55:03) Criminals(55:39) Causes of Crime(56:16) Causes of Violence(57:35) Homelessness(58:27) Yay Trivial Inconveniences(59:08) San Francisco(01:04:07) Closing Down San Francisco(01:05:30) A San Francisco Dispute(01:09:13) Cleaning Up San Francisco(01:13:05) Portland(01:13:15) Those Who Do Not Help Themselves(01:15:15) Solving for the Equilibrium (1)(01:20:15) Solving for the Equilibrium (2)(01:20:43) Lead(01:22:18) Law & Order(01:22:58) Look Out --- First published: November 3rd, 2025 Source: https://www.lesswrong.com/posts/tt9JKubsa8jsCsfD5/crime-and-punishment-1-1 --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

OpenAI is now set to become a Public Benefit Corporation, with its investors entitled to uncapped profit shares. Its nonprofit foundation will retain some measure of control and a 26% financial stake, in sharp contrast to its previous stronger control and much, much larger effective financial stake. The value transfer is in the hundreds of billions, thus potentially the largest theft in human history. I say potentially largest because I realized one could argue that the events surrounding the dissolution of the USSR involved a larger theft. Unless you really want to stretch the definition of what counts this seems to be in the top two. I am in no way surprised by OpenAI moving forward on this, but I am deeply disgusted and disappointed they are being allowed (for now) to do so, including this statement of no action by Delaware and this Memorandum of Understanding with California. Many media and public sources are calling this a win for the nonprofit, such as this from the San Francisco Chronicle. This is mostly them being fooled. They’re anchoring on OpenAI's previous plan to far more fully sideline the nonprofit. This is indeed a big win for [...] ---Outline:(01:38) OpenAI Calls It Completing Their Recapitalization(03:05) How Much Was Stolen?(07:02) The Nonprofit Still Has Lots of Equity After The Theft(10:41) The Theft Was Unnecessary For Further Fundraising(11:45) How Much Control Will The Nonprofit Retain?(23:13) Will These Control Rights Survive And Do Anything?(26:17) What About OpenAI's Deal With Microsoft?(31:10) What Will OpenAI's Nonprofit Do Now?(36:33) Is The Deal Done? --- First published: October 31st, 2025 Source: https://www.lesswrong.com/posts/wCc7XDbD8LdaHwbYg/openai-moves-to-complete-potentially-the-largest-theft-in --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Sometimes the best you can do is try to avoid things getting even worse even faster. Thus, one has to write articles such as ‘Please Do Not Sell B30A Chips to China.’ It's rather crazy to think that one would have to say this out loud. In the same way, it seems not only do we need to say out loud to Not Build Superintelligence Right Now, there are those who say how dare you issue such a statement without knowing how to do so safety, so instead we should build superintelligence without knowing how to do so safety. The alternative is to risk societal dynamics we do not know how to control and that could have big unintended consequences, you say? Yes, well. One good thing to come out of that was that Sriram Krishnan asked (some of) the right questions, giving us the opportunity to try and answer. I also provided updates on AI Craziness Mitigation Efforts from OpenAI and Anthropic. We can all do better here. Tomorrow, I’ll go over OpenAI's ‘recapitalization’ and reorganization, also known as one of the greatest thefts in human history. Compared to what we feared, it looks like we did relatively well [...] ---Outline:(02:01) Language Models Offer Mundane Utility(06:37) Language Models Don't Offer Mundane Utility(12:18) Huh, Upgrades(14:55) On Your Marks(16:40) Choose Your Fighter(22:57) Get My Agent On The Line(24:00) Deepfaketown and Botpocalypse Soon(25:10) Fun With Media Generation(28:42) Copyright Confrontation(28:56) They Took Our Jobs(30:50) Get Involved(30:55) Introducing(32:22) My Name is Neo(34:38) In Other AI News(36:50) Show Me the Money(39:36) One Trillion Dollars For My Robot Army(42:43) One Million TPUs(45:57) Anthropic's Next Move(46:55) Quiet Speculations(53:26) The Quest for Sane Regulations(58:54) The March of California Regulations(01:06:49) Not So Super PAC(01:08:32) Chip City(01:12:52) The Week in Audio(01:13:12) Do Not Take The Bait(01:14:43) Rhetorical Innovation(01:17:02) People Do Not Like AI(01:18:08) Aligning a Smarter Than Human Intelligence is Difficult(01:21:28) Misaligned!(01:23:29) Anthropic Reports Claude Can Introspect(01:30:59) Anthropic Reports On Sabotage Risks(01:34:49) People Are Worried About AI Killing Everyone(01:35:24) Other People Are Not As Worried About AI Killing Everyone(01:38:08) The Lighter Side --- First published: October 30th, 2025 Source: https://www.lesswrong.com/posts/TwbA3zTr99eh2kgCf/ai-140-trying-to-hold-the-line --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

The Chinese and Americans are currently negotiating a trade deal. There are plenty of ways to generate a win-win deal, and early signs of this are promising on many fronts. Since this will be discussed for real tomorrow as per reports, I will offer my thoughts on this one more time. The biggest mistake America could make would be to effectively give up Taiwan, which would be catastrophic on many levels including that Taiwan contains TSMC. I am assuming we are not so foolish as to seriously consider doing this, still I note it. Beyond that, the key thing, basically the only thing, America has to do other than ‘get a reasonable deal overall’ is not be so captured or foolish or both as to allow export of the B30A chip, or even worse than that (yes it can always get worse) allow relaxation of restrictions on semiconductor manufacturing imports. At first I hadn’t heard signs about this. But now it looks like the nightmare of handing China compute parity on a silver platter is very much in play. I disagreed with the decision to sell the Nvidia H20 chips to China, but that [...] ---Outline:(02:01) What It Would Mean To Sell The B30A(07:22) A Note On The 'Tech Stack'(08:32) A Note On Trade Imbalances(09:06) What If They Don't Want The Chips?(10:15) Nvidia Is Going Great Anyway Thank You(11:20) Oh Yeah That Other Thing --- First published: October 29th, 2025 Source: https://www.lesswrong.com/posts/ijYpLexfhHyhM2HBC/please-do-not-sell-b30a-chips-to-china --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

AI chatbots in general, and OpenAI and ChatGPT and especially GPT-4o the absurd sycophant in particular, have long had a problem with issues around mental health. I covered various related issues last month. This post is an opportunity to collect links to previous coverage in the first section, and go into the weeds on some new events in the later sections. A lot of you should likely skip most of the in-the-weeds discussions. What Are The Problems There are a few distinct phenomena we have reason to worry about: Several things that we group together under the (somewhat misleading) title ‘AI psychosis,’ ranging from reinforcing crank ideas or making people think they’re always right in relationship fights to causing actual psychotic breaks. Thebes referred to this as three problem modes: The LLM as a social relation that draws you into madness, as an object relation [...] ---Outline:(00:36) What Are The Problems(03:06) This Week In Crazy(05:05) OpenAI Updates Its Model Spec(09:00) Detection Rates(11:08) Anthropic Says Thanks For The Memories(12:32) Boundary Violations(18:41) A Note On Claude Prompt Injections(20:17) Conclusion --- First published: October 28th, 2025 Source: https://www.lesswrong.com/posts/vrjM8qLKbiAYKAHTa/ai-craziness-mitigation-efforts --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Consider this largely a follow-up to Friday's post about a statement aimed at creating common knowledge around it being unwise to build superintelligence any time soon. Mainly, there was a great question asked, so I gave a few hour shot at writing out my answer. I then close with a few other follow-ups on issues related to the statement. A Great Question To Disentangle There are some confusing wires potentially crossed here but the intent is great. Scott Alexander: I think removing a 10% chance of humanity going permanently extinct is worth another 25-50 years of having to deal with the normal human problems the normal way. Sriram Krishnan: Scott what are verifiable empirical things ( model capabilities / incidents / etc ) that would make you shift that probability up or down over next 18 months? I went through three steps interpreting [...] ---Outline:(00:30) A Great Question To Disentangle(02:20) Scott Alexander Gives a Fast Answer(04:53) Question 1: What events would most shift your p(doom | ASI) in the next 18 months?(13:01) Question 1a: What would get this risk down to acceptable levels?(14:48) Question 2: What would shift the amount that stopping us from creating superintelligence for a potentially extended period would reduce p(doom)?(17:01) Question 3: What would shift your timelines to ASI (or to sufficiently advanced AI, or 'to crazy')?(20:01) Bonus Question 1: Why Do We Keep Having To Point Out That Building Superintelligence At The First Possible Moment Is Not A Good Idea?(22:44) Bonus Question 2: What Would a Treaty On Prevention of Artificial Superintelligence Look Like?--- First published: October 27th, 2025 Source: https://www.lesswrong.com/posts/mXtYM3yTzdsnFq3MA/asking-some-of-the-right-questions --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Building superintelligence poses large existential risks. Also known as: If Anyone Builds It, Everyone Dies. Where ‘it’ is superintelligence, and ‘dies’ is that probably everyone on the planet literally dies. We should not build superintelligence until such time as that changes, and the risk of everyone dying as a result, as well as the risk of losing control over the future as a result, is very low. Not zero, but far lower than it is now or will be soon. Thus, the Statement on Superintelligence from FLI, which I have signed. Context: Innovative AI tools may bring unprecedented health and prosperity. However, alongside tools, many leading AI companies have the stated goal of building superintelligence in the coming decade that can significantly outperform all humans on essentially all cognitive tasks. This has raised concerns, ranging from human economic obsolescence and disempowerment, losses of freedom, civil liberties [...] ---Outline:(02:02) A Brief History Of Prior Statements(03:51) This Third Statement(05:08) Who Signed It(07:27) Pushback Against the Statement(09:05) Responses To The Pushback(12:32) Avoid Negative Polarization But Speak The Truth As You See It--- First published: October 24th, 2025 Source: https://www.lesswrong.com/posts/QzY6ucxy8Aki2wJtF/new-statement-calls-for-not-building-superintelligence-for --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

The big release this week was OpenAI giving us a new browser, called Atlas. The idea of Atlas is that it is Chrome, except with ChatGPT integrated throughout to let you enter agent mode and chat with web pages and edit or autocomplete text, and that will watch everything you do and take notes to be more useful to you later. From the consumer standpoint, does the above sound like a good trade to you? A safe place to put your trust? How about if it also involves (at least for now) giving up many existing Chrome features? From OpenAI's perspective, a lot of that could have been done via a Chrome extension, but by making a browser some things get easier, and more importantly OpenAI gets to go after browser market share and avoid dependence on Google. I’m going to stick with using Claude [...] ---Outline:(02:01) Language Models Offer Mundane Utility(03:07) Language Models Don't Offer Mundane Utility(04:52) Huh, Upgrades(05:24) On Your Marks(10:15) Language Barrier(12:50) From ChatGPT, a Chinese answer to the question about which qualities children should have:(13:30) ChatGPT in English on the same question:(15:17) Choose Your Fighter(17:19) Get My Agent On The Line(18:54) Fun With Media Generation(23:09) Copyright Confrontation(25:19) You Drive Me Crazy(35:31) They Took Our Jobs(44:06) A Young Lady's Illustrated Primer(44:42) Get Involved(45:33) Introducing(47:15) In Other AI News(48:30) Show Me the Money(51:03) So You've Decided To Become Evil(53:18) Quiet Speculations(56:11) People Really Do Not Like AI(57:18) The Quest for Sane Regulations(01:00:55) Alex Bores Launches Campaign For Congress(01:03:33) Chip City(01:10:17) The Week in Audio(01:13:00) Rhetorical Innovation(01:16:17) Don't Take The Bait(01:27:29) Do You Feel In Charge?(01:29:30) Tis The Season Of Evil(01:34:45) People Are Worried About AI Killing Everyone(01:36:00) The Lighter Side--- First published: October 23rd, 2025 Source: https://www.lesswrong.com/posts/qC3M3x2FwiG2Qm7Jj/ai-139-the-overreach-machines --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Some podcasts are self-recommending on the ‘yep, I’m going to be breaking this one down’ level. This was very clearly one of those. So here we go. As usual for podcast posts, the baseline bullet points describe key points made, and then the nested statements are my commentary. If I am quoting directly I use quote marks, otherwise assume paraphrases. Rather than worry about timestamps, I’ll use YouTube's section titles, as it's not that hard to find things via the transcript as needed. This was a fun one in many places, interesting throughout, frustrating in similar places to where other recent Dwarkesh interviews have been frustrating. It gave me a lot of ideas, some of which might even be good. Double click to interact with video AGI Is Still a Decade Away Andrej calls this the ‘decade of agents’ contrary to (among [...] ---Outline:(00:58) AGI Is Still a Decade Away(12:13) LLM Cognitive Deficits(15:31) RL Is Terrible(17:24) How Do Humans Learn?(24:17) AGI Will Blend Into 2% GDP Growth(29:28) ASI(37:38) Evolution of Intelligence and Culture(38:57) Why Self Driving Took So Long(42:06) Future Of Education(48:27) Reactions--- First published: October 21st, 2025 Source: https://www.lesswrong.com/posts/ZBoJaebKFEzxuhNGZ/on-dwarkesh-patel-s-podcast-with-andrej-karpathy --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

We have the classic phenomenon where suddenly everyone decided it is good for your social status to say we are in an ‘AI bubble.’ Are these people short the market? Do not be silly. The conventional wisdom response to that question these days is that, as was said in 2007, ‘if the music is playing you have to keep dancing.’ So even with lots of people newly thinking there is a bubble the market has not moved down, other than (modestly) on actual news items, usually related to another potential round of tariffs, or that one time we had a false alarm during the DeepSeek Moment. So, what's the case we’re in a bubble? What's the case we’re not? My Answer In Brief People get confused about bubbles, often applying that label any time prices fall. So you have to be clear on what [...] ---Outline:(01:17) My Answer In Brief(02:18) Time Sensitive Point of Order: Alex Bores Launches Campaign For Congress, If You Care About AI Existential Risk Consider Donating(04:09) So They're Saying There's a Bubble(05:04) AI Is Atlas And People Worry It Might Shrug(05:35) Can A Bubble Be Common Knowledge?(08:33) Steamrollers, Picks and Shovels(09:27) What Can Go Up Must Sometimes Go Down(11:36) What Can Go Up Quite A Lot Can Go Even More Down(13:17) Step Two Remains Important(15:00) Oops We Might Do It Again(15:49) Derek Thompson Breaks Down The Arguments(17:47) AI Revenues Are Probably Going To Go Up A Lot(20:19) True Costs That Matter Are Absolute Not Relative(21:05) We Are Spending a Lot But Also Not a Lot(22:46) Valuations Are High But Not Super High(23:59) Official GPU Depreciation Schedules Seem Pretty Reasonable To Me(29:14) The Bubble Case Seems Weak(30:53) What It Would Mean If Prices Did Go Down--- First published: October 20th, 2025 Source: https://www.lesswrong.com/posts/rkiBknhWh3D83Kdr3/bubble-bubble-toil-and-trouble --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

As usual when things split, Part 1 is mostly about capabilities, and Part 2 is mostly about a mix of policy and alignment. Table of Contents The Quest for Sane Regulations. The GAIN Act and some state bills. People Really Dislike AI. They would support radical, ill-advised steps. Chip City. Are we taking care of business? The Week in Audio. Hinton talks to Jon Stewart, Klein to Yudkowsky. Rhetorical Innovation. How to lose the moral high ground. Water Water Everywhere. AI has many big issues. Water isn’t one of them. Read Jack Clark's Speech From The Curve. It was a sincere, excellent speech. How One Other Person Responded To This Thoughtful Essay. Some aim to divide. A Better Way To Disagree. Others aim to work together and make things better. Voice Versus Exit. The age old [...] ---Outline:(00:20) The Quest for Sane Regulations(05:56) People Really Dislike AI(12:22) Chip City(13:12) The Week in Audio(13:24) Rhetorical Innovation(20:53) Water Water Everywhere(23:57) Read Jack Clark's Speech From The Curve(28:26) How One Other Person Responded To This Thoughtful Essay(38:43) A Better Way To Disagree(59:39) Voice Versus Exit(01:03:51) The Dose Makes The Poison(01:06:44) Aligning a Smarter Than Human Intelligence is Difficult(01:10:08) You Get What You Actually Trained For(01:15:54) Messages From Janusworld(01:18:37) People Are Worried About AI Killing Everyone(01:22:40) The Lighter SideThe original text contained 1 footnote which was omitted from this narration. --- First published: October 17th, 2025 Source: https://www.lesswrong.com/posts/gCuJ5DabY9oLNDs9B/ai-138-part-2-watch-out-for-documents --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.