DiscoverLessWrong (30+ Karma)“Measuring no CoT math time horizon (single forward pass)” by ryan_greenblatt
“Measuring no CoT math time horizon (single forward pass)” by ryan_greenblatt

“Measuring no CoT math time horizon (single forward pass)” by ryan_greenblatt

Update: 2025-12-26
Share

Description

A key risk factor for scheming (and misalignment more generally) is opaque reasoning ability.
One proxy for this is how good AIs are at solving math problems immediately without any chain-of-thought (CoT) (as in, in a single forward pass).
I've measured this on a dataset of easy math problems and used this to estimate 50% reliability no-CoT time horizon using the same methodology introduced in Measuring AI Ability to Complete Long Tasks (the METR time horizon paper).


Important caveat: To get human completion times, I ask Opus 4.5 (with thinking) to estimate how long it would take the median AIME participant to complete a given problem.
These times seem roughly reasonable to me, but getting some actual human baselines and using these to correct Opus 4.5's estimates would be better.


Here are the 50% reliability time horizon results:



I find that Opus 4.5 has a no-CoT 50% reliability time horizon of 3.5 minutes and that time horizon has been doubling every 9 months.


In an earlier post (Recent LLMs can leverage filler tokens or repeated problems to improve (no-CoT) math performance), I found that repeating the problem substantially boosts performance.
In the above plot, if [...]

---

Outline:

(02:33 ) Some details about the time horizon fit and data

(04:17 ) Analysis

(06:59 ) Appendix: scores for Gemini 3 Pro

(11:41 ) Appendix: full result tables

(11:46 ) Time Horizon - Repetitions

(12:07 ) Time Horizon - Filler

The original text contained 2 footnotes which were omitted from this narration.

---


First published:

December 26th, 2025



Source:

https://www.lesswrong.com/posts/Ty5Bmg7P6Tciy2uj2/measuring-no-cot-math-time-horizon-single-forward-pass


---


Narrated by TYPE III AUDIO.


---

Images from the article:

Two graphs comparing
Two graphs comparing AI model training times with and without repeat/filler, linear and log scale.
Graph showing
Two histograms showing
Graph showing model success probability versus estimated solve time with fitted sigmoid curve titled

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Comments 
loading
In Channel
loading
00:00
00:00
1.0x

0.5x

0.8x

1.0x

1.25x

1.5x

2.0x

3.0x

Sleep Timer

Off

End of Episode

5 Minutes

10 Minutes

15 Minutes

30 Minutes

45 Minutes

60 Minutes

120 Minutes

“Measuring no CoT math time horizon (single forward pass)” by ryan_greenblatt

“Measuring no CoT math time horizon (single forward pass)” by ryan_greenblatt