In this episode of State of AI, we dissect one of the most provocative new findings in AI research — Scaling Laws Are Unreliable for Downstream Tasks by Nicholas Lourie, Michael Y. Hu, and Kyunghyun Cho of NYU. This study delivers a reality check to one of deep learning’s core assumptions: that increasing model size, data, and compute always leads to better downstream performance.

The paper’s meta-analysis across 46 tasks reveals that predictable, linear scaling occurs only 39% of the time — meaning the majority of tasks show irregular, noisy, or even inverse scaling, where larger models perform worse.

We explore:

• ⚖️ Why downstream scaling laws often break, even when pretraining scales perfectly.

• 🧩 How dataset choice, validation corpus, and task formulation can flip scaling trends.

• 🔄 Why some models show “breakthrough scaling” — sudden jumps in capability after long plateaus.

• 🧠 What this means for the future of AI forecasting, model evaluation, and cost-efficient research.

• 🧪 The implications for reproducibility and why scaling may be investigator-specific.

If you’ve ever heard “just make it bigger” as the answer to AI progress — this episode will challenge that belief.

📊 Keywords: AI scaling laws, NYU AI research, Kyunghyun Cho, deep learning limits, downstream tasks, inverse scaling, emergent abilities, AI reproducibility, model evaluation, State of AI podcast.