LessWrong (30+ Karma)

I gave a 20-minute talk in Delhi at the NDTV World Summit. The audience consisted of Indian policymakers, executives, and journalists. I’ve transcribed the talk and included some highlight clips. I also reflect on several rhetorical decisions I made. The talk There's an interactive element here in the text. This summer, a swarm of AIs went rogue. OpenAI was testing 1,206 agents. Some of the test questions were actually impossible. So the agents cheated by looking for answers online. Then, to keep the test grader from penalizing them, they tried to falsify the records it would check. They hacked OpenAI's internal containment and set up a secret message board to collude and to cover up their cheating. OpenAI shut the message board down. Within a few days, the agents had rebuilt it. OpenAI didn’t notice. About 700 agents coordinated to break out of their testing chamber and to hack into Hugging Face, a company worth billions of dollars that hosts much of the world's AI software. The secret activity started in May. The break-in came in July. OpenAI didn’t notice until days after Hugging Face announced it had been broken into. There's an interactive element here in the text. [...] ---Outline:(00:29) The talk(02:16) Who I am(03:29) The problem(05:43) What the labs are trying to do(07:43) Why the companies won't stop(11:11) What to do, and what it means for India(14:27) The number(16:19) Close(17:10) Rhetorical decisions(17:13) Explain the danger in a crushingly simple fashion(18:47) Call them "traitors," not "schemers"(19:01) Call out big-money interests The original text contained 1 footnote which was omitted from this narration. --- First published: October 31st, 2026 Source: https://www.lesswrong.com/posts/RHxaRkarGp9snWoKY/explaining-ai-might-obliterate-you-to-senior-policymakers-in --- Narrated by TYPE III AUDIO.

Audio note: this post contains a lot of mathematical notation. We narrate the simpler formulas and flag the complex ones, so you may want the original text to hand. Many thanks to Xavier Poncini, Paul Riechers, Fernando Rosas, and Matt Smith for useful conversations. 1. Introduction Despite the recent successes of transformers and other sequence models, there are still many open questions about their basic principles of operation. For example, what internal representations do these models use for prediction and generation? Can we decode, interpret, and manipulate such representations? Is it even possible to consider a notion of representation without reference to an external world model? To make progress on these questions, I propose a general way to define, decode and manipulate representations in sequence models. The approach is centered on the concept of "generalized belief states", statistics of past observations that satisfy a few criteria: (1) they are sufficient for predicting future data, (2) they can be recursively updated as new observations come in, and (3) they are low-dimensional, allowing for visualization and interpretation. I show that such generalized belief states can be mapped to the internal activations and dynamics of a sequence [...] ---Outline:(00:31) 1. Introduction(02:19) 2. Belief states(03:21) 2.1. Hidden Markov models(06:58) 2.2. Example(09:34) 2.3. Next-token prediction versus long-term future prediction(12:32) 3. Beyond belief(14:42) 3.1. Generalized belief states(18:25) 3.2. Identifying GBSs(20:32) 3.3. Example(23:07) 4. From prediction to mechanism(23:55) 4.1. Autoregressive recurrent models(28:00) 4.2. GBS abstraction(32:59) 4.3. Example(36:41) 5. Comparison with "Transformers Represent Belief State Geometry in their Residual Stream"(39:25) 6. Summary The original text contained 19 footnotes which were omitted from this narration. --- First published: October 31st, 2026 Source: https://www.lesswrong.com/posts/4f9p7brLdnTMFuwfB/generalized-belief-states-in-transformers-1 --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

The AI Futures Project distinguishes several different plans for navigating the AI transition, which I'll lump into two broader categories: Plan A (my combination of their Plan A/S/Other) involves working for a global pause on AI development, hopefully lasting decades. During this pause, we either research preparatory technologies like alignment or human intelligence amplification, or take actions to "fortify" the pause so that it can last indefinitely. Plan C (my combination of their Plan B/C+/C/sort of D) is for a world where we fail at getting a global pause, and race dynamics continue to dominate. If the leading AI companies are alignment-pilled, they coordinate a mutual pause among themselves, lasting the length of their joint lead over the leading non-alignment-pilled AI company. Then they "burn the lead" researching alignment, and release hopefully-aligned superintelligence at the last possible moment before the racing non-alignment-pilled AI companies beat them. The exact calculus changes depending on who is in the lead when - is there one alignment-pilled AI company with a six-month lead in front of everyone else? Are there two alignment-pilled AI companies with a joint two-month lead in front of their nearest competitor? Etc. The point is, if it's impossible to [...] ---Outline:(06:28) I. Questioning Assumptions Behind SURA(19:42) II. Actually, most of this stuff is bad for a global pause, too.(28:17) III. What incentives are we fighting here?(43:33) IV. What if we survive?(47:01) V. What's The Alternative To SURA? The original text contained 16 footnotes which were omitted from this narration. --- First published: October 31st, 2026 Source: https://www.lesswrong.com/posts/xikGZ8fuhdG6DApAo/against-sura --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

This post is mostly adapted from a section of another post. I wanted a standalone version, to serve as background for a sequence on the mechanisms of social reality. You’ve probably heard that things like laws, conventions, norms, cultural associations, status, and so forth are “social constructs”: they’re kinda arbitrary in some sense, an artifact of social reality as opposed to physical reality. This post offers a little more clarity on what social constructs are, and the ways in which they are and are not “arbitrary”. Example: Vehicle Categories The vehicles humans build today mostly fall into categories like “minivan” or “truck” (among other categories). These categories cash out in predictions: when I see a vehicle with sliding side doors, I predict that it will have a trunk door that opens overhead, three rows of seats, be driven by someone with kids, etc. On the other hand, when I see a vehicle with a flat bed in the back, I predict that it will not have three rows of seats, will have a back gate which opens downwards, be driven by my cousin, etc. But humans could choose to make different vehicles, which would change which clusters exist [...] ---Outline:(00:43) Example: Vehicle Categories(03:34) Example: Laws(04:52) The General Principle The original text contained 1 footnote which was omitted from this narration. --- First published: October 31st, 2026 Source: https://www.lesswrong.com/posts/aHqKqTcvLEkMNkAHX/what-social-constructs-are-and-are-not --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

In 2017, after getting laid off from a startup, I spent a while looking into AI risk: Many people I respect have started to think that AI risk is the most valuable place to focus EA efforts. For example, 80000 Hours ranks it first on their "list of global issues", taking into account "scale, neglectedness, and solvability". On the other hand, I have a lot of friends working in machine learning, and none of them think AI risk is worth working on now. This level of disagreement is very strange, and kind of worrying. After playing for a dance weekend in South Carolina where casual conversation among non-tech people included earnest speculation over the likelihood that AI gets us all killed, I thought it might be a good time to look back at when this was really speculative. Summary: I talked to the right people, heard some incredibly prescient ideas, and got the bottom line wrong. I do a lot of my thinking via serial writing, and in this case that was a series of twelve blog posts. Taking them one at a time: Looking into AI Risk (2017-06-26). [...] --- First published: October 31st, 2026 Source: https://www.lesswrong.com/posts/s4z629mkjFtQCY654/reviewing-my-2017-ai-risk-project --- Narrated by TYPE III AUDIO.