Anton on Twitter: https://twitter.com/atroyn

Moritz on Twitter: https://twitter.com/MoritzW42

Scaling Knowledge: https://scalingknowledge.substack.com

Mentioned Resources

• Unlimiformer: Long-Range Transformers with Unlimited Length Input: arxiv.org/abs/2305.01625


• Memorizing Transformers: arxiv.org/abs/2203.08913


• Chroma: https://www.trychroma.com


• ⁠You and your Research⁠: https://www.cs.virginia.edu/~robins/YouAndYourResearch.html


• Against Method⁠: https://en.wikipedia.org/wiki/Against_Method

Contents:

(00:00 ) Preview

(01:14 ) Introduction

(06:13 ) Edech

(12:12 ) Airchips cost structure

(24:06 ) Techno Optimism

(25:09 ) Convenience vs Enabling Technology

(31:13 ) Chroma and LLM Landscape

(35:37 ) Minecraft Bot

(37:05 ) Knowledge Retrieval

(39:38 ) New Knowledge Retrieval Research and Techniques: (Direct Retrieval Augmentation of Transformers, Conditional Retrieval)

(42:35 ) AI Native Platforms

(45:43 ) Theorem Proving and Generation

(49:40 ) Do LLMs Reason

(52:18 ) Computational Complexity and LessWrong

(53:58 ) Frameworks and Taking Your Work Seriously

(55:53 ) Job Displacement and Jobs Supply vs Demand Side

(58:29 ) The Printing Press