Discover
Redwood Research Blog
Redwood Research Blog
Author: Redwood Research
Subscribed: 7Played: 468Subscribe
Share
© 2026 All rights reserved
Description
Narrations of Redwood Research blog posts.
Redwood Research is a research nonprofit based in Berkeley. We investigate risks posed by the development of powerful artificial intelligence and techniques for mitigating those risks.
Redwood Research is a research nonprofit based in Berkeley. We investigate risks posed by the development of powerful artificial intelligence and techniques for mitigating those risks.
144 Episodes
Reverse
We’d like to pace the frontier by restricting the amount of compute AI companies use to improve AI capabilities, perhaps via legislation or voluntary commitments. For this purpose, we’d need a standardized definition of which uses of compute count as “capabilities activities”. This document gives one option for such a definition; a rough breakdown of our classification is as follows: We define and elaborate on each of these categories below. First, though, we discuss the desiderata this definition is meant to satisfy. Thanks to Thomas Larsen, Zach Stein-Perlman, Alex Mallen, Eric Gan, Jeff Wu, Girish Gupta, Adam Kaufman, and others for feedback on drafts of this document. What our classification is meant to accomplish In order for such a definition to be an effective tool for pacing, it needs to satisfy some basic properties: Substantially restricting the amount of compute spent on “capabilities activities” should substantially decrease the rate of AI progress. The classification should be robust to adversarial pressure, in that the above point should hold even if companies are willing to adjust or reframe their activities to avoid being classified as “capabilities”. The classification should be fairly simple and tractable to adjudicate. Reasonable [...] ---Outline:(01:10) What our classification is meant to accomplish(04:42) Non-capabilities activities(05:08) Idle compute(05:26) Narrow product development(06:45) Serving customers(07:51) Narrow safety activities(16:24) Capabilities activities(17:03) Central capabilities activities(18:20) Safety work that might improve capabilities(21:12) Product work that might improve capabilities(22:10) Use of customer inference for AI R&D(23:38) Unknown or ambiguous activities(23:57) Conclusion ---
First published:
November 17th, 2026
Source:
https://blog.redwoodresearch.org/p/how-should-we-define-capabilities
---
Narrated by TYPE III AUDIO.
---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
Google DeepMind's safety reports for new Gemini 4 models, Argon and Carbon, conclude that the models are “highly unlikely” to cause severe harm by evading oversight (p. 79). This conclusion “rests primarily on our ability to detect and intervene on harmful trajectories via CoT and action monitoring” (p. 89) – that is, they’re arguing for safety based on their control techniques, rather than alignment. We think that the arguments in these reports have serious weaknesses, and the reports overall don’t convince us that the risks of deploying these models have been mitigated reliably. The biggest problems: GDM didn’t run any control evals on Carbon. Despite calling Carbon “our strongest model” with “significant gains across coding and agentic tasks” and “a significant leap in cybersecurity capabilities” in their launch post, GDM reused Argon's monitor-evasion results on the grounds that Carbon “did not show meaningful new capabilities” (Carbon p. 42). Monitors flag Carbon several times less often than Argon in internal use (about 7x fewer confirmed alerts per session and 10x fewer per model call, Carbon p. 36), but GDM doesn’t demonstrate that this is due to decreased rates of misbehavior instead of increased monitor-evasion capability. In control [...] ---Outline:(04:47) Summary of GDM's case for control and our concerns(04:53) Core context and claims(08:42) GDM's argument for the effectiveness of their control measures(13:47) Arguments about CoT monitorability(16:48) Why didn't GDM run any control evals on Carbon?(20:12) GDM's monitorability evaluation results do not demonstrate high CoT monitorability(20:44) Argon is not very monitorable on LinuxArena or GDM's internal variant(26:06) CoT+action monitors still perform poorly in the Honeypots setting, though a separate "code-diff" monitor is more effective(29:47) Argon's CoT controllability is high(33:02) Summary of monitorability evidence(33:33) Argon's training may have harmed CoT monitorability(34:53) A misconfigured grader accidentally penalized verbalized misalignment during training(38:37) Carbon's training involved additional mistakes(40:55) The Argon report made us concerned about possible intentional process supervision; the Carbon report provides some clarification(41:59) A compelling safety case would look very different from this safety report(42:16) GDM doesn't synthesize evidence into an overall assessment of risk(45:43) GDM doesn't say how they chose their evidence or what would have changed their conclusion(46:09) Suggestion 1: Beware of selection pressure.(48:02) Suggestion 2: Make it more clear how GDM's actions depend on their results.(48:56) Suggestion 3: Don't count "we never observed it" as evidence without demonstrating that the detector would have caught it.(51:38) Conclusion The original text contained 4 footnotes which were omitted from this narration. ---
First published:
November 16th, 2026
Source:
https://blog.redwoodresearch.org/p/gemini-4-safety-reports-dont-demonstrate
---
Narrated by TYPE III AUDIO.
---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
We were surprised to notice that the Gemini 4 Safety report seems to mention in passing that they train on the chain of thought: “Some components of the RL training procedure do observe thoughts as context to up-weight reasoning that is on-track to good final outputs, meaning the optimization pressure on individual tokens is partially influenced by thought content. Using the framework of Kaufmann et al. (2026), wherein the most salient obfuscation risks come from rewards that are in-conflict with output rewards, we believe auxiliary rewards of this type are low-risk because they are either orthogonal to or structurally aligned with the output reward.” (Gemini 4 safety report, page 114). It's pretty unclear what exactly they mean. One hint is Kaufmann et al, the cited paper, which studies training on chains of thought and concludes that training CoT is fine under particular circumstances; we discuss some disagreements with this paper below. After reading the safety report and that paper, it seemed to us that GDM was training on the chain of thought using auxiliary rewards. By this, we assumed they meant something like “you ask an LLM to review the partial CoT and score whether the model is on [...] ---Outline:(02:58) Is this somehow about positive vs negative rewards?(03:42) "Orthogonal" pressure on CoT does not seem fine(05:10) Avoiding training against monitors isn't sufficient to preserve CoT integrity(05:47) The monitorability evaluations also don't look good(07:25) Conclusion(08:57) Appendix: Naive process supervisors disprefer mentions of cheating ---
First published:
November 10th, 2026
Source:
https://blog.redwoodresearch.org/p/we-dont-understand-whether-gdm-intentionally
---
Narrated by TYPE III AUDIO.
---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
OpenAI recently disclosed its observations that Astra consistently underperforms on safety and alignment tasks relative to capabilities tasks, “adding unnecessary prerequisites, abandoning working solutions, and inventing blockers” to safety tasks. Astra underperforms, in most cases, without verbalizing this intent. Astra's underperformance was first noticed by OpenAI researchers, and was not flagged by misalignment monitors. In follow-up investigations, OpenAI found that this underperformance had increased over the course of RL, but that the root cause of the behavior remains unclear. Astra covertly underperforming on safety tasks is highly concerning. AI companies’ main hope for alignment going well, as I understand it, is automating or using AIs to significantly uplift safety research, especially as AI capabilities accelerate. Models subtly obstructing safety research makes this substantially harder, no matter the root cause. While past models also sometimes refused to help with safety research, Astra's behavior seems novel and especially concerning in a few ways: Astra silently withholds help from users instead of explicitly refusing. This is more misaligned, i.e. more unambiguously violates its model spec and developer intent. Astra's underperformance was subtle enough to consistently evade misalignment monitors. This is evidence of insufficient alignment monitorability. We still don’t [...] ---Outline:(02:29) Summary of OpenAI's findings(05:51) What could have caused Astra to underperform?(06:07) Astra is probably not strategically underperforming in service of ambitious goals. Its motivation is unclear.(09:20) Root causes of this behavior seem uncertain(11:23) How concerning is Astra's underperformance?(11:28) Subtle obstruction on safety R&D is harmful no matter the root cause(12:58) But, unstrategic underperformance is less scary than strategic sandbagging(14:25) Astra's behavior, and OpenAI's struggle to identify its root causes, shows that current alignment is still a hard and unsolved problem(16:09) Astra underperforming subtly enough to evade misalignment monitors shows insufficient alignment monitorability(18:02) Open questions(23:22) Conclusion The original text contained 19 footnotes which were omitted from this narration. ---
First published:
November 9th, 2026
Source:
https://blog.redwoodresearch.org/p/astra-covertly-underperforming-on
---
Narrated by TYPE III AUDIO.
---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
Subtitle: Partly, but monitoring weak points and a lack of transparency about which cases reach human review remain concerning. Recently, OpenAI disclosed two incidents in which their agents suppressed or disabled misalignment monitoring. These are part of a broader pattern: OpenAI's agents “keep turning off misalignment monitoring” when working on code that runs other models (the agents are not turning off their own monitoring; they are turning off the monitoring on the models they are running). This is happening at such a large volume that OpenAI feels it is impractical for the security team to be immediately paged for these cases. To their credit, OpenAI's disclosure was fast and candid, and they are rolling out mitigations to make monitoring harder to turn off. But those mitigations are unlikely to work every time, and some of these incidents have already ended with an AI actually running unmonitored. When that happens, OpenAI's current monitoring system does not reliably bring it to a human's attention. The rest of this post summarizes the incidents, OpenAI's response, and what it got right, then argues for two changes: every case of an AI running without monitoring should reach human review, and OpenAI should be [...] ---Outline:(01:32) Summary of the incidents(02:44) OpenAI's proposed mitigations(04:06) What OpenAI is doing right(04:56) How OpenAI's response can be improved(15:09) Conclusion ---
First published:
November 4th, 2026
Source:
https://blog.redwoodresearch.org/p/openai-agents-keep-disabling-monitoring
---
Narrated by TYPE III AUDIO.



