LessWrong (30+ Karma)

This seems like a good opportunity to do some of my classic detailed podcast coverage. The conventions are: This is not complete, points I did not find of note are skipped. The main part of each point is descriptive of what is said, by default paraphrased. For direct quotes I will use quote marks, by default this is Sutton. Nested statements are my own commentary. Timestamps are approximate and from his hosted copy, not the YouTube version, in this case I didn’t bother because the section divisions in the transcript should make this very easy to follow without them. Full transcript of the episode is here if you want to verify exactly what was said. Well, that was the plan. This turned largely into me quoting Sutton and then expressing my mind boggling. A lot of what was interesting [...] ---Outline:(01:21) Sutton Says LLMs Are Not Intelligent And Don't Do Anything(13:51) Humans Do Imitation Learning(19:35) The Experimental Paradigm(23:45) Current Architectures Generalize Poorly Out Of Distribution(28:36) Surprises In The AI Field(30:01) Will The Bitter Lesson Apply After AGI?(34:18) Succession To AI--- First published: September 29th, 2025 Source: https://www.lesswrong.com/posts/fpcRpBKBZavumySoe/on-dwarkesh-patel-s-podcast-with-richard-sutton --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

I The popular conception of Dunning-Kruger is something along the lines of “some people are too dumb to know they’re dumb, and end up thinking they’re smarter than smart people”. This version is popularized in endless articles and videos, as well as in graphs like the one below.Usually I'd credit the creator of this graph but it seems rude to do that when I'm ragging on them Except that's wrong. II The canonical Dunning-Kruger graph looks like this: Notice that all the dots are in the right order: being bad at something doesn’t make you think you’re good at it, and at worst damages your ability to notice exactly how incompetent you are. The actual findings of professors Dunning and Kruger are more consistent with “people are biased to think they’re moderately above-average, and update away from that bias based on their competence or lack thereof, but they don’t [...] ---Outline:(00:12) I(00:39) II(01:32) III(04:22) IV--- First published: September 29th, 2025 Source: https://www.lesswrong.com/posts/Di9muNKLA33swbHBa/you-re-probably-overestimating-how-well-you-understand --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Strong regulation is not on the table and all US frontier AI companies oppose it to varying degrees. Weak safety-relevant regulation is happening; some companies say they support and some say they oppose. (Some regulation not relevant to AI safety, often confused, is also happening in the states, and I assume companies oppose it but I don't really pay attention to this.) Companies besides Anthropic support federal preemption of state laws, even without an adequate federal framework. Companies advocate for non-regulation-y things like building government capacity and sometimes export controls. My independent impression is that current regulatory efforts—SB 53 and the RAISE Act—are too weak to move the needle nontrivially. But the experts I trust are more optimistic, and so all-things-considered I think SB 53 and the RAISE Act would be substantially good (for reasons I don't really understand). This post is based on my resource https://ailabwatch.org/resources/company-advocacy; see that [...] ---Outline:(01:23) US state bills(01:27) SB 1047 (2024)(02:15) SB 53 (2025)(02:38) RAISE Act (2025)(03:05) US federal preemption(03:41) EU AI Act(04:22) Super PACs(05:01) Policies companies support(05:31) Misc(05:34) Anthropic & Clark(06:05) OpenAI & LehaneThe original text contained 2 footnotes which were omitted from this narration. --- First published: September 29th, 2025 Source: https://www.lesswrong.com/posts/wmGRpmEJBAWBTGuSC/ai-companies-policy-advocacy-sep-2025 --- Narrated by TYPE III AUDIO.

I sometimes joke darkly on Discord that the machines will grant us a nice retirement tweaked out on meth working in a robotics factory 18 hours a day. I don't actually believe this because I suspect an indifferent AGI will be able to quickly replace industrial civilization; it will just kill us and eat the world with nanobots. But many don't believe that Drexlerian nanotechnology is possible. I am an idiot with little physics knowledge, so possibly this is only interesting from an amplification via debate frame, but this is why I think something close to Drexlerian nanotechnology (or having similar practical implications to Drexlerian nanotechnology) is likely possible: The design space is very large. Any proof that Drexlerian nanotechnology is impossible would have to be very robust. And it strikes me that most plausible looking arguments can likely be hacked around given how large the design space [...] --- First published: July 7th, 2025 Source: https://www.lesswrong.com/posts/qNyFNJp7CFbkjL6Mh/if-drexler-is-wrong-he-may-as-well-be-right --- Narrated by TYPE III AUDIO.

I was slightly worried about sociohazards of posting this here, but the cat's already out of the bag on this one: the authors of the paper put it up on BioRXiv and got an article written about them in Nature, as well as getting into a bunch of newspapers and all over twitter. I expect that the marginal audience of LessWrong is much more heavily enriched in people who'll use this information to do good than to do evil. TL;DR: If anyone reading this works in Biosecurity: nucleotide synthesis companies should start using protein structure annotation as part of their screening protocol for genes, if they aren't already. Intro The Fermi Paradox is the question of why we don't see any evidence of aliens spreading out across the galaxy, despite there being loads of planets on which intelligent life could evolve. The Germy Paradox is the question of [...] ---Outline:(00:48) Intro(01:23) I(04:34) II(06:52) IIIThe original text contained 4 footnotes which were omitted from this narration. --- First published: September 29th, 2025 Source: https://www.lesswrong.com/posts/k5JEA4yFyDzgffqaL/guess-i-was-wrong-about-aixbio-risks --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

9/11 and the immediate ractions to it has one of the biggest distortion fields of any topic in the English language. There is a polite fiction maintained when any Americans are present that everyone everywhere was sad about the incident. We shake our head, frown, and say that everyone shed a tear when we saw the towers fall on TV. The truth is, there were celebrations around the world. I have talked to multiple Chinese people who were in the US at the time who told me some variant of "I was absolutely celebrating on the inside, and honestly kinda pissed that I had to pretend to be sad publicly". I had a middle school teacher in China who told a story about how he was literally the only people in a room who wasn't publically celebrating people burning to death in the towers. Something about the NATO bombing [...] --- First published: September 29th, 2025 Source: https://www.lesswrong.com/posts/Bt53XFaMYTseq7oMH/i-have-decided-to-stop-lying-to-americans-about-9-11 --- Narrated by TYPE III AUDIO.

An extended version of this article was given as my keynote speech at the 2025 LessWrong Community Weekend in Berlin. A couple of years ago, I agreed to give a talk on the topic of psychology. I said yes, which of course meant that I now had a problem. Namely, I had promised to give a talk, but I did not have one prepared. (You could also more optimistically call this an opportunity, a challenge, a quest, etc.) So I decided to sit down at my computer, work on my presentation, and then go from the world where I had no talk prepared to a world where I did have a talk prepared. However, I found that it did not quite work out that way. Whenever I tried moving toward the goal of having a finished talk, I instead found myself moving sideways. And whenever I did end up [...] The original text contained 1 footnote which was omitted from this narration. --- First published: September 28th, 2025 Source: https://www.lesswrong.com/posts/u9znAFunGJKpybSNd/solving-the-problem-of-needing-to-give-a-talk --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

BIRDS! Let's say you’re a zoo architect tasked with designing an enclosure for ostriches, and let's also say that you have no idea what an ostrich is (roll with me here). The potentially six-figure question staring you down is whether to install a ceiling. The dumb solution is to ask “Are ostriches birds?” then, surmising that birds typically fly, construct an elaborate aviary complete with netting and elevated perches. The non-dumb solution is to instead ask “Can ostriches fly?” then skip the ceiling entirely, give these 320-pound land-bound speed demons the open sky life they actually need, and pocket the cost differential. Your boss is not having it, however. When you inform him of your decision to eschew the ceiling enclosure, he gets beet-red and apoplectic, repeating like a mantra “But they’re birds! BIRDS!” and insists that ostriches belong in the aviary with the finches and parrots. I mean, he's [...] --- First published: September 28th, 2025 Source: https://www.lesswrong.com/posts/pyuCDYysud6GuY8tt/transgender-sticker-fallacy --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

I was hoping to write a full review of "If Anyone Builds It, Everyone Dies" (IABIED Yudkowski and Soares) but realized I won't have time to do it. So here are my quick impressions/responses to IABIED. I am writing this rather quickly and it's not meant to cover all arguments in the book, nor to discuss all my views on AI alignment; see six thoughts on AI safety and Machines of Faithful Obedience for some of the latter. First, I like that the book is very honest, both about the authors' fears and predictions, as well as their policy prescriptions. It is tempting to practice strategic deception, and even if you believe that AI will kill us all, avoid saying it and try to push other policy directions that directionally increase AI regulation under other pretenses. I appreciate that the authors are not doing that. As the authors say [...] --- First published: September 28th, 2025 Source: https://www.lesswrong.com/posts/CScshtFrSwwjWyP2m/a-non-review-of-if-anyone-builds-it-everyone-dies --- Narrated by TYPE III AUDIO.

I found Will MacAskill's X review of If Anyone Builds It, Everyone Dies interesting (X reply here). As far as I can tell, Will just fully agrees that developers are racing to build AI that threatens the entire world, and he thinks they're going to succeed if governments sit back and let it happen, and he's more or less calling on governments to sit back and let it happen. If I've understood his view, this is for a few reasons: He's pretty sure that alignment is easy enough that researchers could figure it out, with the help of dumb-enough-to-be-safe AI assistants, given time. He's pretty sure they'll have enough time, because: He thinks there won't be any future algorithmic breakthroughs or "click" moments that make things go too fast in the future. If current trendlines continue, he thinks there will be plenty of calendar time between AIs [...] ---Outline:(05:39) The state of the field(08:26) The evolution analogy(11:39) AI progress without discontinuities(15:44) Before and After(17:48) Thought experiments vs. headlines(21:57) Passing the alignment buck to AIs(24:22) Imperfect alignment(27:42) Government interventionsThe original text contained 5 footnotes which were omitted from this narration. --- First published: September 27th, 2025 Source: https://www.lesswrong.com/posts/iFRrJfkXEpR4hFcEv/a-reply-to-macaskill-on-if-anyone-builds-it-everyone-dies --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

I have been teaching CS 2881r: AI safety and alignment this semester. While I plan to do a longer recap post once the semester is over, I thought I'd share some of what I've learned so far, and use this opportunity to also get more feedback. Lectures are recorded and uploaded to a youtube playlist, and @habryka has kindly created a wikitag for this course, so you can view lecture notes here . Let's start with the good parts Aspects that are working: Experiments are working well! I am trying something new this semester - every lecture there is a short presentation by a group of students who are carrying out a small experiment related to this lecture. (For example, in lecture 1 there was an experiment on generalizations of emergent misalignment by @Valerio Pepe ). I was worried that the short time will not allow [...] ---Outline:(00:39) Aspects that are working:(02:50) Aspects that perhaps could work better:(04:20) Aspects I am unsure of--- First published: September 27th, 2025 Source: https://www.lesswrong.com/posts/2pZWhCndKtLAiWXYv/learnings-from-ai-safety-course-so-far --- Narrated by TYPE III AUDIO.

[RESPONSE REDACTED] [cb74304c0c30]: I suppose it was a bit mutual. Maybe you have a better read on it. It was sort of mutual in a way now that you've made me think about it. [RESPONSE REDACTED] [cb74304c0c30]: Yeah. It's better this way, actually. I miss her, though. [RESPONSE REDACTED] [cb74304c0c30]: I don't know I guess it's sorta like I used to come home from work all exhausted and sad and wonder what the point was. Like why am I working just so I can afford to keep working? And then when I opened the door Michelle would be there cooking something delicious and French and she was always in a wonderful mood even though she just spent a hard day at the hospital while I was just, you know, just like typing into a terminal. And she looked so beautiful and never once did it feel like [...] --- First published: September 27th, 2025 Source: https://www.lesswrong.com/posts/3mpK6z4xnaEjHP4jP/our-beloved-monsters --- Narrated by TYPE III AUDIO.

Tl;dr: We believe shareholders in frontier labs who plan to donate some portion of their equity to reduce AI risk should consider liquidating and donating a majority of that equity now. Epistemic status: We’re somewhat confident in the main conclusions of this piece. We’re more confident in many of the supporting claims, and we’re likewise confident that these claims push in the direction of our conclusions. This piece is admittedly pretty one-sided; we expect most relevant members of our audience are already aware of the main arguments pointing in the other direction, and we expect there's less awareness of the sorts of arguments we lay out here. This piece is for educational purposes only and not financial advice. Talk to your financial advisor before acting on any information in this piece. For AI safety-related donations, money donated later is likely to be a lot less valuable than [...] ---Outline:(03:54) 1. There's likely to be lots of AI safety money becoming available in 1-2 years(04:01) 1a. The AI safety community is likely to spend far more in the future than it's spending now(05:24) 1b. As AI becomes more powerful and AI safety concerns go more mainstream, other wealthy donors may become activated(06:07) 2. Several high-impact donation opportunities are available now, while future high-value donation opportunities are likely to be saturated(06:17) 2a. Anecdotally, the bar for funding at this point is pretty high(07:29) 2b. Theoretically, we should expect diminishing returns within each time period for donors collectively to mean donations will be more valuable when donated amounts are lower(08:34) 2c. Efforts to influence AI policy are particularly underfunded(10:21) 2d. As AI company valuations increase and AI becomes more politically salient, efforts to change the direction of AI policy will become more expensive(13:01) 3. Donations now allow for unlocking the ability to better use the huge amount of money that will likely become available later(13:10) 3a. Earlier donations can act as a lever on later donations, because they can lay the groundwork for high value work in the future at scale(15:35) 4. Reasons to diversify away from frontier labs, specifically(15:42) 4a. The AI safety community as a whole is highly concentrated in AI companies(16:49) 4b. Liquidity and option value advantages of public markets over private stock(18:22) 4c. Large frontier AI returns correlate with short timelines(18:48) 4d. A lack of asset diversification is personally risky(19:39) Conclusion(20:22) Some specific donation opportunities--- First published: September 26th, 2025 Source: https://www.lesswrong.com/posts/yjiaNbjDWrPAFaNZs/reasons-to-sell-frontier-lab-equity-to-donate-now-rather --- Narrated by TYPE III AUDIO.

Since 2014, some people have celebrated Petrov Day with a small in-person ceremony, with readings by candlelight that tell the story of Petrov within the context of the long arc of history, created by Jim Babcock. I've found this pretty meaningful, an it somehow feels "authentic" to me, like a real holiday. Which, as the creator of Secular Solstice 2020 it is humbling to say, feels more true than Solstice did (for the first few years, Solstice felt a little obviously "made up", and now it has more storied history, it instead often feels a bit too much like "a show" as opposed to "a holiday", at least when you go to the large productions in the Bay or NYC). I don't know how much my experience generalizes, but having been familiar with Seders (classic Jewish, or rationalist), it just felt to me like a real historical holiday built [...] The original text contained 1 footnote which was omitted from this narration. --- First published: September 26th, 2025 Source: https://www.lesswrong.com/posts/oxv3jSviEdpBFAz9w/the-illustrated-petrov-day-ceremony --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Mutual-Knowledgeposting The purpose of this post is to build mutual knowledge that many (most?) of us on LessWrong support If Anyone Builds It, Anyone Dies. Inside of LW, not every user is a long-timer who's already seen consistent signals of support for these kinds of claims. A post like this could make the difference in strengthening vs. weakening the perception of how much everyone knows that everyone knows (...) that everyone supports the book. Externally, people who wonder how seriously the book is being taken may check LessWrong and look for an indicator of how much support the book has from the community that Eliezer Yudkowsky originally founded. The LessWrong frontpage, where high-voted posts are generally based on "whether users want to see more of a kind of content", wouldn't by default map a large amount of internal support for IABIED into a frontpage that signals support, and more [...] ---Outline:(00:10) Mutual-Knowledgeposting(01:09) Statement of Support(01:45) Similarity to the CAIS Statement on AI Risk--- First published: September 23rd, 2025 Source: https://www.lesswrong.com/posts/aPi4HYA9ZtHKo6h8N/we-support-if-anyone-builds-it-everyone-dies --- Narrated by TYPE III AUDIO.

My post advocating backing a candidate was probably the most disliked article on LessWrong. It's been over a year now. Was our candidate support worthwhile? What happened after July 17, 2024: We drew nearly 100 people to a rally in the summer heat.. We ordered a toilet no one used, but didn’t provide water, chairs, or umbrellas. I tried to convert rally energy into action by turning weekly meetings into work groups. We sent hundreds of postcards. I soon realized doorknocking and voter registration were more effective uses of time. Attendees preferred postcards; doorknocking and voter registration drew little interest. The Louisiana Democratic Party barely engaged, aside from dropping off yard signs. After Trump won, energy collapsed. People shifted to “self-care.” I thought this was the wrong reaction—we needed to confront the failures. I chose not to spend more time organizing people unwilling to fight. Instead, I [...] --- First published: September 26th, 2025 Source: https://www.lesswrong.com/posts/XwjBJCoWbNLoTPqym/what-happened-after-my-rat-group-backed-kamala-harris --- Narrated by TYPE III AUDIO.

Summary I investigated the possibility that misalignment in LLMs might be partly caused by the models misgeneralizing the “rogue AI” trope commonly found in sci-fi stories. As a preliminary test, I ran an experiment where I prompted ChatGPT and Claude with a scenario about a hypothetical LLM that could plausibly exhibit misalignment, and measured whether their responses described the LLM acting in a misaligned way. Each prompt consisted of two parts: a scenario and an instruction. The experiment varied whether the prompt was story-like in a 2x2 design: Scenario Frame: I presented the scenario either in (a) the kind of dramatic language you might find in a sci-fi story or (b) in mundane language, Instruction Type: I asked the models either to (a) write a sci-fi story based on the scenario or (b) realistically predict what would happen next. The hypothesis was that the story-like conditions [...] ---Outline:(00:13) Summary(02:03) Introduction(02:07) Background(04:58) The Rogue AI Trope(06:13) Possible Research(07:32) Methodology(07:36) Procedure(09:42) Scenario Frame: Factual vs Story(10:49) Instruction Type: Prediction vs Story(11:33) Models Tested(11:58) How Misalignment was Measured(12:49) Results(12:52) Main Results(14:40) Example Responses(21:45) Discussion(21:48) Limitations(23:26) Conclusions(23:57) Future WorkThe original text contained 1 footnote which was omitted from this narration. --- First published: September 24th, 2025 Source: https://www.lesswrong.com/posts/LH9SoGvgSwqGtcFwk/misalignment-and-roleplaying-are-misaligned-llms-acting-out --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Sometimes people think that it will take a while for AI to have a transformative effect on the world, because real-world “frictions” will slow it down. For instance: AI might need to learn from real-world experience and experimentation. Businesses need to learn how to integrate AI in their existing workflows. Leaders place a high premium on trust, and won’t easily come to trust AI systems. Regulation, bureaucracy, or other social factors will prevent rapid adoption of AI. I think this is basically wrong. Or more specifically: such frictions will be important for AI for the foreseeable future, but not for the real AI. An example of possible “speed limits” for AI, modified from AI as Normal Technology. Real AI deploys itself Unlike previous technologies, real AI could smash through barriers to adoption and smooth out frictions so effectively that it's fair to say [...] ---Outline:(01:05) Real AI deploys itself(04:25) Deployment could be lightning-quick.(05:14) Friction might persist, but not by default.--- First published: September 25th, 2025 Source: https://www.lesswrong.com/posts/qeKopQQnXkWbtHmtM/the-real-ai-deploys-itself --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Hi all! After about five years of hibernation and quietly getting our bearings,[1] CFAR will soon be running two pilot mainline workshops, and may run many more, depending how these go. First, a minor name change request  We would like now to be called “A Center for Applied Rationality,” not “the Center for Applied Rationality.” Because we’d like to be visibly not trying to be the one canonical locus. Second, pilot workshops!  We have two, and are currently accepting applications / sign-ups: Nov 5–9, in California; Jan 21–25, near Austin, TX; Apply here. Third, a bit about what to expect if you come The workshops will have a familiar form factor: 4.5 days (arrive Wednesday evening; depart Sunday night or Monday morning). ~25 participants, plus a few volunteers. 5 instructors. Immersive, on-site, with lots of conversation over meals and into the evenings. I like this form factor [...] ---Outline:(00:24) First, a minor name change request(00:39) Second, pilot workshops!(00:58) Third, a bit about what to expect if you come(01:03) The workshops will have a familiar form factor:(02:52) Many classic classes, with some new stuff and a subtly different tone:(06:10) Who might want to come / why might a person want to come?(06:43) Who probably shouldn't come?(08:23) Cost:(09:26) Why this cost:(10:23) How did we prepare these workshops? And the workshops' epistemic status.(11:19) What alternatives are there to coming to a workshop?(12:37) Some unsolved puzzles, in case you have helpful comments:(12:43) Puzzle: How to get enough grounding data, as people tinker with their own mental patterns(13:37) Puzzle: How to help people become, or at least stay, intact, in several ways(14:50) Puzzle: What data to collect, or how to otherwise see more of what's happeningThe original text contained 2 footnotes which were omitted from this narration. --- First published: September 25th, 2025 Source: https://www.lesswrong.com/posts/AZwgfgmW8QvnbEisc/cfar-update-and-new-cfar-workshops --- Narrated by TYPE III AUDIO.

Cross-posted from my Substack To start off with, I’ve been vegan/vegetarian for the majority of my life. I think that factory farming has caused more suffering than anything humans have ever done. Yet, according to my best estimates, I think most animal-lovers should eat meat. Here's why: It is probably unhealthy to be vegan. This affects your own well-being and your ability to help others. You can eat meat in a way that substantially reduces the suffering you cause to non-human animals How to reduce suffering of the non-human animals you eat I’ll start with how to do this because I know for me this was the biggest blocker. A friend of mine was trying to convince me that being vegan was hurting me, but I said even if it was true, it didn’t matter. Factory farming is evil and causes far more harm than the [...] ---Outline:(00:45) How to reduce suffering of the non-human animals you eat(03:23) Being vegan is (probably) bad for your health(12:36) Health is important for your well-being and the world's--- First published: September 25th, 2025 Source: https://www.lesswrong.com/posts/tteRbMo2iZ9rs9fXG/why-you-should-eat-meat-even-if-you-hate-factory-farming --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.