P(Doom) Estimates Shouldn't Inform Policy??
Description
Princeton Comp Sci Ph.D. candidate Sayash Kapoor co-authored a blog post last week with his professor Arvind Narayanan called "AI Existential Risk Probabilities Are Too Unreliable To Inform Policy".
While some non-doomers embraced the arguments, I see it as contributing nothing to the discourse besides demonstrating a popular failure mode: a simple misunderstanding of the basics of Bayesian epistemology.
I break down Sayash's recent episode of Machine Learning Street Talk point-by-point to analyze his claims from the perspective of the one true epistemology: Bayesian epistemology.
00:00 Introduction
03:40 Bayesian Reasoning
04:33 Inductive vs. Deductive Probability
05:49 Frequentism vs Bayesianism
16:14 Asteroid Impact and AI Risk Comparison
28:06 Quantification Bias
31:50 The Extinction Prediction Tournament
36:14 Pascal's Wager and AI Risk
40:50 Scaling Laws and AI Progress
45:12 Final Thoughts
My source material is Sayash's episode of Machine Learning Street Talk: https://www.youtube.com/watch?v=BGvQmHd4QPE
I also recommend reading Scott Alexander’s related post: https://www.astralcodexten.com/p/in-continued-defense-of-non-frequentist
Sayash's blogpost that he was being interviewed about is called "AI existential risk probabilities are too unreliable to inform policy": https://www.aisnakeoil.com/p/ai-existential-risk-probabilities
Follow Sayash: https://x.com/sayashk
This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit lironshapira.substack.com
United States
