DiscoverMachine Learning Street Talk (MLST)Sayash Kapoor - How seriously should we take AI X-risk? (ICML 1/13)
Sayash Kapoor - How seriously should we take AI X-risk? (ICML 1/13)

Sayash Kapoor - How seriously should we take AI X-risk? (ICML 1/13)

Update: 2024-07-28
Share

Description

How seriously should governments take the threat of existential risk from AI, given the lack of consensus among researchers? On the one hand, existential risks (x-risks) are necessarily somewhat speculative: by the time there is concrete evidence, it may be too late. On the other hand, governments must prioritize — after all, they don’t worry too much about x-risk from alien invasions.




MLST is sponsored by Brave:


The Brave Search API covers over 20 billion webpages, built from scratch without Big Tech biases or the recent extortionate price hikes on search API access. Perfect for AI model training and retrieval augmentated generation. Try it now - get 2,000 free queries monthly at brave.com/api.




Sayash Kapoor is a computer science Ph.D. candidate at Princeton University's Center for Information Technology Policy. His research focuses on the societal impact of AI. Kapoor has previously worked on AI in both industry and academia, with experience at Facebook, Columbia University, and EPFL Switzerland. He is a recipient of a best paper award at ACM FAccT and an impact recognition award at ACM CSCW. Notably, Kapoor was included in TIME's inaugural list of the 100 most influential people in AI.




Sayash Kapoor


https://x.com/sayashk


https://www.cs.princeton.edu/~sayashk/




Arvind Narayanan (other half of the AI Snake Oil duo)


https://x.com/random_walker




AI existential risk probabilities are too unreliable to inform policy


https://www.aisnakeoil.com/p/ai-existential-risk-probabilities




Pre-order AI Snake Oil Book


https://amzn.to/4fq2HGb




AI Snake Oil blog


https://www.aisnakeoil.com/




AI Agents That Matter


https://arxiv.org/abs/2407.01502




Shortcut learning in deep neural networks


https://www.semanticscholar.org/paper/Shortcut-learning-in-deep-neural-networks-Geirhos-Jacobsen/1b04936c2599e59b120f743fbb30df2eed3fd782




77% Of Employees Report AI Has Increased Workloads And Hampered Productivity, Study Finds


https://www.forbes.com/sites/bryanrobinson/2024/07/23/employees-report-ai-increased-workload/




TOC:


00:00:00 Intro


00:01:57 How seriously should we take Xrisk threat?


00:02:55 Risk too unrealiable to inform policy


00:10:20 Overinflated risks


00:12:05 Perils of utility maximisation


00:13:55 Scaling vs airplane speeds


00:17:31 Shift to smaller models?


00:19:08 Commercial LLM ecosystem


00:22:10 Synthetic data


00:24:09 Is AI complexifying our jobs?


00:25:50 Does ChatGPT make us dumber or smarter?


00:26:55 Are AI Agents overhyped?


00:28:12 Simple vs complex baselines


00:30:00 Cost tradeoff in agent design


00:32:30 Model eval vs downastream perf


00:36:49 Shortcuts in metrics


00:40:09 Standardisation of agent evals


00:41:21 Humans in the loop


00:43:54 Levels of agent generality


00:47:25 ARC challenge

Comments 
00:00
00:00
x

0.5x

0.8x

1.0x

1.25x

1.5x

2.0x

3.0x

Sleep Timer

Off

End of Episode

5 Minutes

10 Minutes

15 Minutes

30 Minutes

45 Minutes

60 Minutes

120 Minutes

Sayash Kapoor - How seriously should we take AI X-risk? (ICML 1/13)

Sayash Kapoor - How seriously should we take AI X-risk? (ICML 1/13)

Machine Learning Street Talk (MLST)