DiscoverAI Safety Fundamentals: AlignmentDeep Double Descent
Deep Double Descent

Deep Double Descent

Update: 2024-06-17
Share

Description

We show that the double descent phenomenon occurs in CNNs, ResNets, and transformers: performance first improves, then gets worse, and then improves again with increasing model size, data size, or training time. This effect is often avoided through careful regularization. While this behavior appears to be fairly universal, we don’t yet fully understand why it happens, and view further study of this phenomenon as an important research direction.


Source:

https://openai.com/research/deep-double-descent


Narrated for AI Safety Fundamentals by Perrin Walker of TYPE III AUDIO.

---

A podcast by BlueDot Impact.

Learn more on the AI Safety Fundamentals website.

Comments 
Top Podcasts
The Best New Comedy Podcast Right Now – June 2024The Best News Podcast Right Now – June 2024The Best New Business Podcast Right Now – June 2024The Best New Sports Podcast Right Now – June 2024The Best New True Crime Podcast Right Now – June 2024The Best New Joe Rogan Experience Podcast Right Now – June 20The Best New Dan Bongino Show Podcast Right Now – June 20The Best New Mark Levin Podcast – June 2024
In Channel
Eliciting Latent Knowledge

Eliciting Latent Knowledge

2024-06-1701:00:27

Deep Double Descent

Deep Double Descent

2024-06-1708:27

Chinchilla’s Wild Implications

Chinchilla’s Wild Implications

2024-06-1724:57

Intro to Brain-Like-AGI Safety

Intro to Brain-Like-AGI Safety

2024-06-1701:02:10

Gradient Hacking: Definitions and Examples

Gradient Hacking: Definitions and Examples

2024-06-1709:15

An Investigation of Model-Free Planning

An Investigation of Model-Free Planning

2024-06-1708:11

Discovering Latent Knowledge in Language Models Without Supervision

Discovering Latent Knowledge in Language Models Without Supervision

2024-06-1737:09

Toy Models of Superposition

Toy Models of Superposition

2024-06-1741:43

Imitative Generalisation (AKA ‘Learning the Prior’)

Imitative Generalisation (AKA ‘Learning the Prior’)

2024-06-1718:14

ABS: Scanning Neural Networks for Back-Doors by Artificial Brain Stimulation

ABS: Scanning Neural Networks for Back-Doors by Artificial Brain Stimulation

2024-06-1716:08

Least-To-Most Prompting Enables Complex Reasoning in Large Language Models

Least-To-Most Prompting Enables Complex Reasoning in Large Language Models

2024-06-1716:08

Two-Turn Debate Doesn’t Help Humans Answer Hard Reading Comprehension Questions

Two-Turn Debate Doesn’t Help Humans Answer Hard Reading Comprehension Questions

2024-06-1716:39

Low-Stakes Alignment

Low-Stakes Alignment

2024-06-1713:56

Empirical Findings Generalize Surprisingly Far

Empirical Findings Generalize Surprisingly Far

2024-06-1711:32

Worst-Case Thinking in AI Alignment

Worst-Case Thinking in AI Alignment

2024-05-2911:35

How to Get Feedback

How to Get Feedback

2024-05-1207:30

Public by Default: How We Manage Information Visibility at Get on Board

Public by Default: How We Manage Information Visibility at Get on Board

2024-05-1209:50

How to Succeed as an Early-Stage Researcher: The “Lean Startup” Approach

How to Succeed as an Early-Stage Researcher: The “Lean Startup” Approach

2024-04-2315:16

Become a Person who Actually Does Things

Become a Person who Actually Does Things

2024-04-1705:14

Working in AI Alignment

Working in AI Alignment

2024-04-1401:08:44

loading
Google Play
Download from Google Play
Castbox
Download from App Store
usUnited States
00:00
00:00
x

0.5x

0.8x

1.0x

1.25x

1.5x

2.0x

3.0x

Sleep Timer

Off

End of Episode

5 Minutes

10 Minutes

15 Minutes

30 Minutes

45 Minutes

60 Minutes

120 Minutes

Deep Double Descent

Deep Double Descent

BlueDot Impact