OLMoE and the hidden simplicity in training better foundation models
Description
Ai2 released OLMoE, which is probably our "best" model yet relative to its peers, but not much has changed in the process.
This is AI generated audio with Python and 11Labs.
Source code: https://github.com/natolambert/interconnects-tools
Original post: https://www.interconnects.ai/p/olmoe-and-building-better-llms
00:00 OLMoE and the hidden simplicity in training better foundation models
02:04 Frontier model team compute allocations
04:19 De-risking training complexity
06:40 On organizational complexity
09:05 Compounding improvements -- the key to building better language models
Fig 1: https://huggingface.co/datasets/natolambert/interconnects-figures/resolve/main/olmoe/img_005.png
Fig 2: https://huggingface.co/datasets/natolambert/interconnects-figures/resolve/main/olmoe/img_007.png
Fig 3: https://huggingface.co/datasets/natolambert/interconnects-figures/resolve/main/olmoe/img_009.png
Fig 4: https://huggingface.co/datasets/natolambert/interconnects-figures/resolve/main/olmoe/img_011.png
Fig 5: https://huggingface.co/datasets/natolambert/interconnects-figures/resolve/main/olmoe/img_028.png
Fig 6: https://huggingface.co/datasets/natolambert/interconnects-figures/resolve/main/olmoe/img_030.png
Fig 7: https://huggingface.co/datasets/natolambert/interconnects-figures/resolve/main/olmoe/img_032.png
Get full access to Interconnects at www.interconnects.ai/subscribe
![[Article Voiceover] AI Safety's Crux: Culture vs. Capitalism](https://substackcdn.com/feed/podcast/48206/post/149851748/5dbaf463a3193f0320d3e30ae332a118.jpg "[Article Voiceover] AI Safety's Crux: Culture vs. Capitalism")
![[Article Voiceover] Llama 3.2 Vision and Molmo: Foundations for the multimodal open-source ecosystem](https://s3.castbox.fm/de/53/16/9e08bcb80bb30117167ed5823009231ff8_scaled_v1_400.jpg "[Article Voiceover] Llama 3.2 Vision and Molmo: Foundations for the multimodal open-source ecosystem")
![[Article Voiceover] Reverse engineering OpenAI's o1](https://s3.castbox.fm/c7/01/d3/7fa03400effff0109644d490f3c2fecf4f_scaled_v1_400.jpg "[Article Voiceover] Reverse engineering OpenAI's o1")
United States
