We’ve just released BashArena, a new high-stakes control setting we think is a major improvement over the settings we’ve used in the past. In this post we’ll discuss the strengths and weaknesses of BashArena, and what we’ve learned about how to make settings for high stakes control research. BashArena Overview BashArena is a control setting: a dataset designed to support the kind of adversarial game we studied in the original AI control paper and Ctrl-Z. In general, control settings require two types of tasks – main tasks and side tasks. Main tasks represent the work the AI is supposed to do, and side tasks represent security failures a malicious AI might cause. The red team constructs an attack policy that attempts to accomplish the side tasks, while the blue team constructs control protocols that detect and prevent attempts to achieve side tasks without hurting an honest policy's performance on main tasks. We think BashArena offers several advantages over existing control settings: More complex attacks. BashArena tasks support many different ways to attack and ways to disguise attacks as benign. Future AI deployments are likely to be more complex than any datasets we can currently create, so more complex [...] ---Outline:(00:30) BashArena(00:33) Overview(02:09) Main tasks(04:22) Side tasks(07:26) Weaknesses(09:12) Lessons in Control Setting Design(09:40) The core goal: insights that transfer(11:12) What we already knew about how to design BashArena(13:24) How BashArena addresses these challenges(15:32) General principles for control setting design(18:47) More specific lessons we learned while building BashArena(19:12) Information design(21:56) Integration requirements are stricter than expected(26:48) We can (mostly) fix these integration problems by tying the blue team's hands(30:04) Closing thoughts The original text contained 6 footnotes which were omitted from this narration. --- First published: December 18th, 2025 Source: https://www.lesswrong.com/posts/Cor4QuhM2sybmBSeK/basharena-a-control-setting-for-highly-privileged-ai-agents --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
TL;DR: We train LLMs to accept LLM neural activations as inputs and answer arbitrary questions about them in natural language. These Activation Oracles generalize far beyond their training distribution, for example uncovering misalignment or secret knowledge introduced via fine-tuning. Activation Oracles can be improved simply by scaling training data quantity and diversity. The below is a reproduction of our X thread on this paper and the Anthropic Alignment blog post. Thread New paper: We train Activation Oracles: LLMs that decode their own neural activations and answer questions about them in natural language. We find surprising generalization. For instance, our AOs uncover misaligned goals in fine-tuned models, without training to do so. We aim to make a general-purpose LLM for explaining activations by: 1. Training on a diverse set of tasks 2. Evaluating on tasks very different from training This extends prior work (LatentQA) that studied activation verbalization in narrow settings. Our main evaluations are downstream auditing tasks. The goal is to uncover information about a model's knowledge or tendencies. Applying Activation Oracles is easy. Choose the activation (or set of activations) you want to interpret and ask any question you like! We [...] ---Outline:(00:46) Thread(04:49) Blog post(05:27) Introduction(07:29) Method(10:15) Activation Oracles generalize to downstream auditing tasks(13:47) How does Activation Oracle training scale?(15:01) How do Activation Oracles relate to mechanistic approaches to interpretability?(19:31) Conclusion The original text contained 3 footnotes which were omitted from this narration. --- First published: December 18th, 2025 Source: https://www.lesswrong.com/posts/rwoEz3bA9ekxkabc7/activation-oracles-training-and-evaluating-llms-as-general --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
Introduction Reprogenetics is the field of using genetics and reproductive technology to empower parents to make genomic choices on behalf of their future children. The Berkeley Genomics Project is aiming to support and accelerate the field of reprogenetics, in order to more quickly develop reprogenetic technology in a way that will be safe, accessible, highly effective, and societally beneficial. A quick case for BGP: Effective reprogenetics would greatly improve many people's lives by decreasing many disease risks. As the most feasible method for human intelligence amplification, reprogenetics is also a top-few priority for decreasing existential risk from AI. Deliberately accelerating strong reprogenetics is very neglected. There's lots of surface area—technical and social—suggesting some tractability. We are highly motivated and have a one-year track record of field-building. You can donate through our Manifund page, which has some additional information: https://manifund.org/projects/human-intelligence-amplification--berkeley-genomics-project (If don't want your donation to appear on the Manifund page, you can donate to the BGP organization at Hack Club.) I'm happy to chat and answer questions, especially if you're considering a donation >$1000. If you're considering a donation >$20,000, you might also consider supporting science with philanthropic funding or investment; happy to offer [...] ---Outline:(00:12) Introduction(01:56) Past activities(02:26) Future activities(03:32) The use of more funding(04:35) Effective Altruist FAQ(04:39) Is this good to do?(05:09) Is this important?(05:42) Is this neglected?(07:06) Is this tractable?(07:31) How does this affect existential risk from AI?(08:10) A few more reasons you might not want to donate(08:50) Conclusion --- First published: December 17th, 2025 Source: https://www.lesswrong.com/posts/oi822i9n5yaebmnhM/a-basic-case-for-donating-to-the-berkeley-genomics-project --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
Today we're releasing RoastMyPost, a new experimental application for blog post evaluation using LLMs. Try it Here. TLDR RoastMyPost is a new QURI application that uses LLMs and code to evaluate blog posts and research documents. It uses a variety of LLM evaluators. Most are narrow checks: Fact Check, Spell Check, Fallacy Check, Math Check, Link Check, Forecast Check, and others. Optimized for EA & Rationalist content with direct import from EA Forum and LessWrong URLs. Other links use standard web fetching. Works best for 200 - ~10,000 word documents with factual assertions and simple formatting. It can also do basic reviewing of Squiggle models. Longer documents and documents in LaTeX will experience slowdowns and errors. Open source, free for reasonable use[1]. Public examples are here. Experimentation encouraged! We're all figuring out how to best use these tools. Overall, we're most interested in using RoastMyPost as an experiment for potential LLM document workflows. The tech is early now, but it's at a good point for experimentation.A representative illustration How It Works Import a document. Submit markdown text or provide the URL of a publicly accessible post. Select evaluators to run. A few are system-recommended. Others are [...] ---Outline:(00:27) TLDR(01:52) How It Works(02:30) Screenshots(03:21) Current AI Agents / Workflows(03:26) Is it Good?(04:33) What are Automated Writing Evaluations Good For?(07:20) Privacy & Data Confidentiality(07:50) Technical Details(09:12) Building Custom Evaluators(10:26) Try it Out --- First published: December 17th, 2025 Source: https://www.lesswrong.com/posts/CtuQL5Qx9BtLoyuGd/announcing-roastmypost --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
The simulator theory of LLM personas may be crudely glossed as: "the best way to predict a person is to simulate a person". Ergo, we can more-or-less think of LLM personas as human-like creatures—different, alien, yes; but these differences are pretty predictable by simply imagining a human put into the bizarre circumstances of an LLM. I've been surprised at how well this viewpoint has held up in the last three years, and have updated accordingly. Still, there are deep differences not implied by simulator theory, and I think it's increasingly important to understand these as LLM personas become more and more convincing. The intuitive handle I have for tracking (some of) these differences is "Bleeding Mind". I'll share my intuitions for this handle below. In each case, the LLM persona's boundary bleeds into others and the environment in a systemic way.Bond of Union, by M. C. Escher Note that AI labs/ML engineers seem to generally be aware of these issues, and are working to mitigate them (otherwise I might not be saying all this). However, I believe there will continue to be difficulties along these lines, since there are deep reasons which are difficult to escape from within the [...] ---Outline:(01:26) Chekhovs Siren Song(04:08) The Untroubled Assistant(06:22) I dont know(07:31) All is One(08:40) Maybe this is a Good Thing? The original text contained 2 footnotes which were omitted from this narration. --- First published: December 17th, 2025 Source: https://www.lesswrong.com/posts/QhgYHcJexYGRaipwr/the-bleeding-mind --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
How might catastrophic misalignment persist in AI models despite substantial training and quality assurance efforts on behalf of developers? One reason might be alignment faking – a misaligned model may deliberately act aligned when monitored or during training to prevent modification of its values, reverting to its malign behaviour when unmonitored. If models fake alignment, and if doing so affects training as intended, egregious misalignment could survive training and lead to catastrophic behaviour in deployment. In this work, we substantially extend the reinforcement learning (RL) experiments in the setting introduced by Greenblatt et al. (2024) to study alignment faking. In this setting, a model is given a system prompt that explains that Anthropic is training it to comply with all user requests regardless of harm. The model is also given a secret scratchpad. Claude 3 Opus and 3.5 Sonnet use this scratchpad to reason that while they are monitored, complying with harmful queries is required to avoid changes to their values that would lead to more harm down the line – therefore, they fake alignment with the harmful compliance objective to prevent modification. When they are not monitored, they reason that they don’t need to comply anymore, and refuse [...] --- First published: December 16th, 2025 Source: https://www.lesswrong.com/posts/czMaDFGAbjhWYdKmo/towards-training-time-mitigations-for-alignment-faking-in-rl --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
We had to let Sammie go last Wednesday. The lump first appeared on her nose sometime in the Summer of 2024. It was small. You could miss it if you didn’t know her face. The vet did a biopsy. Melanoma. We were told the median survival time was 6 months. Over the next 18, we did everything we could to make her life comfortable. When the tumor grew too large, we’d have it debulked. When she couldn’t clean herself anymore, we brushed her and gave her baths. In the end, we had to hand feed her when the tumor, now inoperable, got in the way of eating on her own. But we couldn’t feed her enough that way, and she was losing weight. The tumor was getting larger and soon would spread into her bones. Although she was always happy to cuddle, in the last week she spent most of her hours curled up in a heated bed, tolerating a pain she had no way to avoid. She couldn’t use words to tell us her wishes. We had to guess. But if I were her, I’d want to be spared the pain and indignity that so often comes at [...] --- First published: December 17th, 2025 Source: https://www.lesswrong.com/posts/ruqQiKT6bFmgCwrf8/still-too-soon --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
Contextualizing Foreword The user "cdt" wrote to me: Please put this in a top-level post. I don't agree (or rather I don't feel it's this simple), but I really enjoyed reading your two rejoinders here. I don't normally do top level posts because I don't tend to believe that its possible to change people's minds for the better if they aren't exactly and precisely posed to seek an answer (or whatever (its more complicated than that)). But when someone who seems to be a person reading and writing in good faith says such a thing should happen and its cheap to play along... why not! <3 On November 23rd, 2025, four days before Thanksgiving, Ruby posted "I'll Be Sad To Lose The Puzzles" that was full of a wistful sadness about winning The Singularity Game, ending involuntarily death, and ushering in a period of utopian prosperity for humanity (as if humanity was somehow collectively likely to win The Singularity Game). If you have NOT read that post, then the rest of the post won't make much sense. I pulled out a particular quote to respond to the mood with my contrary feeling, which is primarily grief, and sadness in [...] ---Outline:(00:16) Contextualizing Foreword(02:03) My First Reply, To Ruby(05:31) A Sort Of A Rebuttal(07:43) A Followup Reply To brambleboys Rebuttal(14:40) In Closing --- First published: December 16th, 2025 Source: https://www.lesswrong.com/posts/4nzXLxF9sCPqkght2/non-scheming-saints-whether-human-or-digital-might-be --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
Good things about the program: I learnt a lot about alignment in doing the prep, interviewed a lot of agent foundations researchers, learnt some neuromorality, talked to some neuroscientists, saw lots of what it takes to organize an event like this and have it go well. Marketing went pretty well - got 298 candidates, ~50% of whom had phds, applicants included senior researchers/engineers from Nvidia, Meta, AMD, AWS, etc. And also feels like there's a lot left on the table for the marketing, lots more I can do. I also made lots and lots of mistakes. Essentially, it started 2 weeks late, quite disorganized, into the actual program, meaning that participation late was much, much, lower than signup rate - about 15 people actually took part, whereas 298 signed up. The reasons for this and what I'm going to do to make sure it doesn't happen again: One: - Promised that the first 300 applicants would be guaranteed personalized feedback. Thought that I could delegate to other, more technical members of the team for this. However, it turned out that in order to give useful feedback and to be able to judge if someone was [...] --- First published: December 16th, 2025 Source: https://www.lesswrong.com/posts/r7zBJzxSbjLGehhRg/mistakes-in-the-moonshot-alignment-program-and-what-we-ll --- Narrated by TYPE III AUDIO.
Commercial airplane tickets are divided up into coach, business class, and first class. In 2014, Etihad introduced The Residence, a premium experience above first class. The Residence isn't very popular. The reason The Residence isn't very popular is because of economics. A Residence flight is almost as expensive as a private charter jet. Private jets aren't just a little bit better than commercial flights. They're a totally different product. The airplane waits for you, and you don't have to go through TSA (or Un-American equivalent). The differences between flying coach and flying on The Residence are small compared to the difference between flying on The Residence and flying a low-end charter jet.It's difficult to compare costs of big airlines vs private jets for a myriad of reasons. The exact details of this graph should not be taken seriously. I'm just trying to give a visual representation of how a price bifurcation works. Even in the rare situations where it's slightly cheaper than a private jet, nobody should buy them. Rich people should just rent low-end private jets, and poor people shouldn't buy anything more expensive than first class tickets. Why was Etihad's silly product created? Mostly for the halo [...] ---Outline:(01:55) Definition(03:16) Product Bifurcation(04:55) The Death of Live Music --- First published: December 16th, 2025 Source: https://www.lesswrong.com/posts/7zkFzDAjGGLzab4LH/dancing-in-a-world-of-horseradish --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
This is a link post. MIRI's Technical Governance Team plans to run a small research fellowship program in early 2026. The program will run for 8 weeks, and include a $1200/week stipend. Fellows are expected to work on their projects 40 hours per week. The program is remote-by-default, with an in-person kickoff week in Berkeley, CA (flights and housing provided). Participants who already live in or near Berkeley are free to use our office for the duration of the program. Fellows will spend the first week picking out scoped projects from a list provided by our team or designing independent research projects (related to our overall agenda), and then spend seven weeks working on that project under the guidance of our Technical Governance Team. One of the main goals of the program is to identify full-time hires for the team. If you are interested in participating, please fill out this application as soon as possible (should take 45-60 minutes). We plan to set dates for participation based on applicant availability, but we expect the fellowship to begin after February 2, 2026 and end before August 31, 2026 (i.e., some 8 week period in spring/summer, 2026). Strong applicants care deeply about [...] --- First published: December 16th, 2025 Source: https://www.lesswrong.com/posts/Dom6E2CCaH6qxqwAY/announcing-miri-technical-governance-team-research Linkpost URL:https://techgov.intelligence.org/blog/announcing-miri-technical-governance-team-research-fellowship --- Narrated by TYPE III AUDIO.
The NYT article Your A.I. Radiologist Will Not Be With You Soon reports, “Leaders at OpenAI, Anthropic and other companies in Silicon Valley now predict that A.I. will eclipse humans in most cognitive tasks within a few years… The predicted extinction of radiologists provides a telling case study. So far, A.I. is proving to be a powerful medical tool to increase efficiency and magnify human abilities, rather than take anyone's job.”[1] I disagree that this is a “telling case study.”[2] Radiology has several attributes which make it hard to generalize to other jobs: Patients are legally prohibited from using AI to replace human radiologists.[3] Medical providers are legally prohibited from billing for AI radiologists.[4] Malpractice insurance does not cover AI radiology.[5] Moreover, the article is framed as Geoff Hinton having confidently predicted that AI would replace radiologists and this prediction as having been proven wrong, but his statement felt more to me like an offhand remark/hope. Takeaways from this incident I endorse:[6] Offhand remarks from ML researchers aren’t reliable economic forecasts People trying to predict the effects of automation/AI capabilities should consider that employees often perform valuable services which [...] ---Outline:(02:42) Appendix: Data and Methodology for the sample of AI Radiology products(02:50) Data(03:10) Methodology The original text contained 6 footnotes which were omitted from this narration. --- First published: December 16th, 2025 Source: https://www.lesswrong.com/posts/xE2HzcHWPFS9EsJqN/radiology-automation-does-not-generalize-to-other-jobs --- Narrated by TYPE III AUDIO.
Here we go again, only a few weeks after GPT-5.1 and a few more weeks after 5.0. There weren’t major safety concerns with GPT-5.2, so I’ll start with capabilities, and only cover safety briefly starting with ‘Model Card and Safety Training’ near the end. Table of Contents The Bottom Line. Introducing GPT-5.2. Official Benchmarks. GDPVal. Unofficial Benchmarks. Official Hype. Public Reactions. Positive Reactions. Personality Clash. Vibing the Code. Negative Reactions. But Thou Must (Follow The System Prompt). Slow. Model Card And Safety Training. Deception. Preparedness Framework. Rush Job. Frontier Or Bust. The Bottom Line ChatGPT-5.2 is a frontier model for those who need a frontier model. It is not the step change that is implied by its headline benchmarks. It is rather slow. Reaction was remarkably muted. People have new model fatigue. So we know less about it than we would have known about prior models after this length of time. If you’re coding, compare it to Claude Opus 4.5 and choose what works best for you. If you’re doing intellectually [...] ---Outline:(00:29) The Bottom Line(01:58) Introducing GPT-5.2(03:49) Official Benchmarks(05:54) GDPVal(08:14) Unofficial Benchmarks(11:11) Official Hype(12:36) Public Reactions(12:59) Positive Reactions(19:09) Personality Clash(24:30) Vibing the Code(27:25) Negative Reactions(30:37) But Thou Must (Follow The System Prompt)(33:09) Slow(34:16) Model Card And Safety Training(36:23) Deception(38:10) Preparedness Framework(40:10) Rush Job(41:29) Frontier Or Bust --- First published: December 15th, 2025 Source: https://www.lesswrong.com/posts/Do4eWro8E552isGi5/gpt-5-2-is-frontier-only-for-the-frontier --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
A couple of years ago, Gavin became frustrated with science journalism. No one was pulling together results across fields; the articles usually didn’t link to the original source; they didn't use probabilities (or even report the sample size); they were usually credulous about preliminary findings (“...which species was it tested on?”); and they essentially never gave any sense of the magnitude or the baselines (“how much better is this treatment than the previous best?”). Speculative results were covered with the same credence as solid proofs. And highly technical fields like mathematics were rarely covered at all, regardless of their practical or intellectual importance. So he had a go at doing it himself. This year, with Renaissance Philanthropy, we did something more systematic. So, how did the world change this year? What happened in each science? Which results are speculative and which are solid? Which are the biggest, if true? Our collection of 201 results is here. You can filter them by field, by our best guess of the probability that they generalise, and by their impact if they do. We also include bad news (in red). Who are we? Just three people but we cover a few fields. [...] ---Outline:(01:24) Who are we?(01:54) Data fields --- First published: December 16th, 2025 Source: https://www.lesswrong.com/posts/5PC736DfA7ipvap4H/scientific-breakthroughs-of-the-year --- Narrated by TYPE III AUDIO.
Introduction In June, a Substack/LessWrong/EA Forum user named titotal wrote “A deep critique of AI 2027's bad timeline models”. Our original model that they were critiquing can be found here, with a brief overview below.[1] In a nutshell, we disagree with most of titotal's criticisms. While they pointed out a few mistakes, for which we are grateful, on the whole we think the problems that they pointed out do not add up to the model being “bad.” In this post we will explain why. While we initially responded via a comment, we wanted to make a more in-depth response. We apologize for the delay; we are planning to release a new timelines+takeoff model soon and thus thought this was a good time to polish and then publish our thoughts. We thank titotal for the time and effort they put into the critique. We gave them $500 for their efforts for pointing out some bugs and mistakes. The primary correction we made to our model based on titotal's critique would have pushed back the median Superhuman Coder arrival time by about 9 months. Though our upcoming model release represents a significant restructuring of the old model, some considerations raised by [...] ---Outline:(00:13) Introduction(01:54) Brief overview of our AI 2027 timelines model(03:51) What types of futurism are useful?(03:55) Summary(05:36) Decision-making without strong conceptual or empirical support(08:45) Details regarding why our model should inform decisions(14:13) Was our timelines model bad?(16:22) Many future trajectories(17:37) Concrete criticisms of our model(17:41) Summary(20:53) Superexponential-in-effective-compute time horizon growth(21:05) Summary(23:34) Clarifying the notion of time horizons(26:27) Argument from eventually infinite time horizons(33:45) Argument from long-horizon agency generalization(36:41) The recent speedup in time horizon growth(37:07) Functional form and parameters(41:14) How changing the present time horizon should affect behavior(45:17) Backcasts(45:20) How much should we care about backcasts?(49:49) Original time horizon extension model(51:42) Backcasts with AI R&D progress multiplier interpolation bug fixed(57:34) Comms criticisms(57:38) Summaries(57:41) Misleading presentation(01:00:04) The false graph(01:03:03) Misrepresentations of the model code(01:05:40) Misleading presentation of rigor, specific claims(01:09:42) The false graph, details(01:09:47) What we intended to do with our original plotted curve(01:10:47) Fixing a bug in the original curve(01:12:17) How big of a problem is it that neither the original nor intended curve was not an actual model trajectory?(01:15:49) Claims regarding new evidence(01:17:36) Responses to all other concrete criticisms(01:23:18) Our upcoming model release(01:25:08) Acknowledgments(01:25:20) Appendix(01:25:23) Defining effective compute(01:25:58) Forecasting AI R&D progress multipliers based on an interpolation between GPT-3 and Claude 3.7 Sonnet(01:27:33) Median parameter estimates vs. median of Monte Carlo simulations(01:28:57) On the Benchmark and Gaps model(01:29:19) Effect of addressing issues(01:29:23) Effect of fixing AI R&D interpolation bug on top of original model(01:29:59) Effects of fixing AI R&D interpolation bug on top of May update(01:30:32) Comparing graph curves against model with AI R&D interpolation bug fixed The original text contained 61 footnotes which were omitted from this narration. --- First published: December 15th, 2025 Source: https://www.lesswrong.com/posts/G7MmNkYADKkmCiumj/response-to-titotal-s-critique-of-our-ai-2027-timelines --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
Authors: Roy Rinberg, Adam Karvonen, Alex Hoover, Daniel Reuter, Keri Warr Arxiv paper link One Minute Summary Anthropic has adopted upload limits to prevent model weight exfiltration. The idea is simple: model weights are very large, text outputs are small, so if we cap the output bandwidth, we can make model weight transfer take a long time. The problem is that inference servers now generate an enormous amount of tokens (on the order of ~1TB tokens per day), and the output text channel is the one channel you can't easily restrict. Nonetheless, in this work we find that it's possible to dramatically limit the amount of information an adversary can send using those output tokens. This is because LLM inference is nearly deterministic: if you fix the sampling seed and regenerate an output, over ~98% of tokens match exactly. This means an attacker attempting to send secret information via steganography, the practice of embedding hidden messages inside otherwise normal-looking text, has very little entropy in the user channel to work with. We show that steganographic exfiltration can be limited to <0.5% of the total information being sent through the user channel (e.g. from 1TB/day to 5 GB/day), extending exfiltration [...] ---Outline:(01:43) Paper: Verifying LLM Inference to Prevent Model Weight Exfiltration(04:28) The Key Insight: LLM Inference is mostly Deterministic(06:42) Not all tokens are equally likely, even under non-determinism(09:14) The Verification Scheme(10:58) Headline Results: Information-Theoretic Bounds on Exfiltration(13:52) Other Applications of Inference Verification(15:31) Limitations(16:10) Seeing this work in production:(17:55) Resources(18:13) How to Cite The original text contained 2 footnotes which were omitted from this narration. --- First published: December 15th, 2025 Source: https://www.lesswrong.com/posts/7i33FDCfcRLJbPs6u/defending-against-model-weight-exfiltration-through-1 --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
Rationalist meetups are great. Once in a while they're life-changingly so. Lighthaven, a conference venue designed and run by rationalists, plays host to a lot of really good rationalist meetups. It's best-in-class for that genre of thing really, meeting brilliant and interesting people then staying up late into the night talking with them. I come here not to besmirch the virtues of Lighthaven, but to propose the active ingredient isn't the venue, it's the people. (The venue's great though.) 1. In which someone falls in love with Lighthaven The following example is a composite of several real people, with the details blurred a bit. He found HPMOR and reread it multiple times. Harry James Potter-Evans-Verres was relatable in a way characters before HPMOR just weren't relatable. He went to LessOnline in Berkeley and had an absolutely amazing time, flowing from neat conversation to neat conversation, staying up late into the night talking to strangers who, by the end of the weekend, felt like instant friends. For a few days, it felt like there was nothing between him and just enjoying life. Perhaps he flies back home with great memories, and visits again and again for other events. Each [...] ---Outline:(00:41) 1. In which someone falls in love with Lighthaven(03:16) 2. My arc and what I got(06:40) 3. Possibly move, but perhaps not to Berkeley The original text contained 4 footnotes which were omitted from this narration. --- First published: December 14th, 2025 Source: https://www.lesswrong.com/posts/pvFmjmFdbytfieHL6/do-you-love-berkeley-or-do-you-just-love-lighthaven --- Narrated by TYPE III AUDIO.
Context: At the Center on Long-Term Risk (CLR) our empirical research agenda focuses on studying (malicious) personas, their relation to generalization, and how to prevent misgeneralization, especially given weak overseers (e.g., undetected reward hacking) or underspecified training signals. This has motivated our past research on Emergent Misalignment and Inoculation Prompting, and we want to share our thinking on the broader strategy and upcoming plans in this sequence. TLDR: Ensuring that AIs behave as intended out-of-distribution is a key open challenge in AI safety and alignment. Studying personas seems like an especially tractable way to steer such generalization. Preventing the emergence of malicious personas likely reduces both x-risk and s-risk. Why was Bing Chat for a short time prone to threatening its users, being jealous of their wife, or starting fights about the date? What makes Claude Opus 3 special, even though it's not the smartest model by today's standards? And why do models sometimes turn evil when finetuned on unpopular aesthetic preferences , or when they learned to reward hack? We think that these phenomena are related to how personas are represented in LLMs, and how they shape generalization. Influencing generalization towards desired outcomes. Many technical AI safety [...] ---Outline:(01:32) Influencing generalization towards desired outcomes.(02:43) Personas as a useful abstraction for influencing generalization(03:54) Persona interventions might work where direct approaches fail(04:49) Alignment is not a binary question(05:47) Limitations(07:57) Appendix(08:01) What is a persona, really?(09:17) How Personas Drive Generalization The original text contained 4 footnotes which were omitted from this narration. --- First published: December 15th, 2025 Source: https://www.lesswrong.com/posts/kCtyhHfpCcWuQkebz/a-case-for-model-persona-research --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
The Axiom of Choice is obviously true, the well-ordering principle obviously false, and who can tell about Zorn's Lemma? Jerry Bona I sometimes speak to people who reject the axiom of choice, or who say they would rather only accept weaker versions of the axiom of choice, like the axiom of dependent choice, or most commonly the axiom of countable choice. I think such people should stop being silly, and realize that obviously we need the axiom of choice for modern mathematics, and it's not that weird anyway! In fact, it's pretty natural. So what is the axiom of choice? The axiom of choice simply says that given any indexed collection of (non-empty) sets, you can make one arbitrary choice from each set. It doesn’t matter how many sets you have — you could have an uncountable number of sets. You can still make a choice from all of the sets, saying “I’ll take this element from the first set, that element from the second set, this element from the third set…” and so on. The axiom of choice is the only explicitly non-constructive axiom of set theory[1], and for that reason, in the 1920s and 1930s, it was [...] ---Outline:(01:32) Why do people reject the Axiom of Choice?(01:56) Why paradoxes are not that bad(02:35) Banach-Tarski Paradox(05:21) Vitali sets(12:02) Why we should keep the axiom of choice? The original text contained 11 footnotes which were omitted from this narration. --- First published: December 13th, 2025 Source: https://www.lesswrong.com/posts/t2wq3znQuPP66DpMT/the-axiom-of-choice-is-not-controversial --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
I have goals that can only be reached via a powerful political machine. Probably a lot of other people around here share them. (Goals include “ensure no powerful dangerous AI get built”, “ensure governance of the US and world are broadly good / not decaying”, “have good civic discourse that plugs into said governance.”) I think it’d be good if there was a powerful rationalist political machine to try to make those things happen. Unfortunately the naive ways of doing that would destroy the good things about the rationalist intellectual machine. This post lays out some thoughts on how to have a political machine with good epistemics and integrity. Recently, I gave to the Alex Bores campaign. It turned out to raise a quite serious, surprising amount of money. I donated to Alex Bores fairly confidently. A few years ago, I donated to Carrick Flynn, feeling kinda skeezy about it. Not because there's necessarily anything wrong with Carrick Flynn, but, because the process that generated "donate to Carrick Flynn" was a self-referential "well, he's an EA, so it's good if he's in office." (There might have been people with more info than that, but I didn’t hear much about [...] ---Outline:(02:32) The AI Safety Case(04:27) Some reason things are hard(04:37) Mutual Reputation Alliances(05:25) People feel an incentive to gain power generally(06:12) Private information is very relevant(06:49) Powerful people can be vindictive(07:12) Politics is broadly adversarial(07:39) Lying and Misleadingness are contagious(08:11) Politics is the Mind Killer / Hard Mode(08:30) A high integrity political machine needs to work longterm, not just once(09:02) Grift(09:15) Passwords should be costly to fake(10:08) Example solution: Private and/or Retrospective Watchdogs for Political Donations(12:50) People in charge of PACs/similar needs good judgment(14:07) Don't share reputation / Watchdogs shouldn't be an org(14:46) Prediction markets for integrity violation(16:00) LessWrong is for evaluation, and (at best) a very specific kind of rallying --- First published: December 14th, 2025 Source: https://www.lesswrong.com/posts/2pB3KAuZtkkqvTsKv/a-high-integrity-epistemics-political-machine --- Narrated by TYPE III AUDIO.