Neural intel Pod

Neural Intel Podcast EpisodeMoE Giants: Decoding the 670 Billion Parameter Showdown Between DeepSeek V3 and Mistral LargeThis week on Neural Intel, we dive deep into the architectural blueprints of two colossal Mixture-of-Experts (MoE) models: DeepSeek V3 (673B/671B) and Mistral 3 Large (675B/673B). We explore the configurations that define these massive language models, noting their shared traits, such as an embedding dimension of 7,168 and a vocabulary size of 129K. Both architectures employ a FeedForward (SwiGLU) module, and the initial three blocks use a dense FFN with a hidden size of 18,432 instead of the MoE layer.The core of the discussion focuses on how each model utilizes its MoE layer, both of which contain 128 experts. We contrast the resource allocation and expert frequency: DeepSeek V3/R1 is configured to activate one shared expert plus six additional experts per token (1 shared + 6 experts active per token), resulting in only 37B active parameters per inference step. In contrast, Mistral 3 Large activates one shared expert plus four additional experts per token (1 shared + 4 experts active per token), leading to 39B active parameters per inference step.We also analyze other crucial architectural differences visible in their configuration files, including the intermediate hidden layer dimensions—2,048 for DeepSeek V3/R1 versus 4,096 for Mistral 3 Large. Join us as we dissect how these subtle parameter choices—affecting multi-head latent attention, expert distribution, and shared experts—impact overall efficiency and performance in the race to build the most capable and resourceful large language models.

In this episode of the Neural Intel podcast, we go under the hood of GLM-4.7, the newest native agentic LLM from Z.AI. Released on December 22, 2025, this model represents a massive 41% reasoning improvement over its predecessor, GLM-4.6.We discuss the strategic decision to release an incremental 4.7 update rather than jumping to version 5.0, focusing on how Z.AI has optimized tool orchestration and multilingual coding. Our deep dive covers:• The "Thinking" Revolution: How Preserved Thinking maintains reasoning across multi-turn dialogues to reduce information loss.• Benchmark Wars: Analyzing its 84.9 score on LiveCodeBench and how it stacks up against GPT-5.2 and Gemini 3 Pro.• Hardware and Deployment: What it takes to run a 358B parameter model locally using vLLM or SGLang.Join the conversation: 🐦 Follow us on X/Twitter: @neuralintelorg 🌐 Read the full technical report at: neuralintel.org

In this deep-dive episode, Neural Intel goes behind the data of the Medmarks v0.1 benchmark suite, led by Sophont and the MedARC community. While previous benchmarks like MultiMedQA have "saturated," Medmarks introduces MedXpertQA, a reasoning-heavy task that currently pushes even the strongest frontier models to their limits.We examine the technical nuances of the study:• Thinking vs. Instruct: How reasoning post-training creates a "Pareto improvement" in medical accuracy.• The Efficiency Gap: Why open-weight models like Qwen3 match frontier accuracy but require 5x to 6x the token volume to get there.• Order Bias: The surprising discovery that even frontier models like Grok 4 can be "tripped up" simply by shuffling the order of multiple-choice answers.• Medical Specialization: Does a "medical-tuned" model like MedGemma actually outperform a generalist giant?.Join us as we discuss how these benchmarks are doubling as reinforcement learning environments to train the next generation of digital clinicians.

In this episode of Neural Intel, we break down Andrej Karpathy’s "2025 LLM Year in Review," exploring the massive paradigm shifts that redefined artificial intelligence over the last year. From the technical evolution of the training stack to the cultural phenomenon of "vibe coding," 2025 marked the transition from simple chatbots to "summoned ghosts" and autonomous agents,.Key topics we cover:• The Rise of RLVR: Discover why Reinforcement Learning from Verifiable Rewards (RLVR) has replaced RLHF as the de facto final stage of LLM training, enabling models to develop "reasoning" strategies by solving math and code puzzles,.• Jagged Intelligence: Karpathy argues that AI is not a "growing animal" but a "summoned ghost". We discuss why LLMs can be polymath geniuses in one moment and confused grade-schoolers the next—a phenomenon known as jagged performance.• The "Vibe Coding" Revolution: Learn how programming has shifted to natural language, making code "ephemeral, malleable, and discardable". We look at how this empowers non-coders and allows professionals to build custom tools like tokenizers in minutes.• LLM Agents & GUIs: Why Claude Code and Gemini Nano banana represent a new frontier where AI lives on your local computer and communicates through visual interfaces rather than just text consoles,.• The Death of Benchmarks: As labs "benchmax" through synthetic data and RLVR, Karpathy warns that crushing benchmarks no longer equates to reaching AGI,.As Karpathy notes, the industry has likely realized less than 10% of the potential of current LLM capabilities. Whether you're a developer or an AI enthusiast, these shifts represent a "terraforming" of the software landscape.--------------------------------------------------------------------------------To understand the shift from RLHF to RLVR, think of it as the difference between a student trying to please a teacher (who might be inconsistent or biased) versus a student solving a Rubik's cube. With the cube, the success is objectively verifiable, allowing the student to practice and improve for much longer periods without needing constant human feedback.

This episode dives into neural_net_checklist, the indispensable PyTorch toolkit designed to automate the crucial diagnostic process for training complex neural networks. Inspired by Andrei Karpathy's seminal blog post, "A Recipe for Training Neural Networks," this repository transforms a manual debugging guide into a set of programmatic assertions, saving developers significant time and allowing them to focus on model development.For ML Insiders: Stop guessing why your training loops are failing. This tool provides instant verification of key health indicators, ensuring your model is initialized correctly and your data flow is robust.Key Concepts Covered in This Episode:• Initialization Health Checks: We explore how the tool verifies the model's setup, including asserting that the loss at initialization is within the expected range for balanced classification tasks (e.g., close to −log(1/Nclasses​)). It also checks if the model is well-calibrated at initialization, ensuring initial predictions are uniformly distributed.• Data Flow Integrity: Learn about the critical assertions that verify how data moves through your model, specifically:    ◦ Forward and Backward Batch Independence: Checks whether computations (and gradients) for one sample are unaffected by others in the batch, a crucial check which often requires replacing norm layers (like LayerNorm or BatchNorm) with Identity during the test, since batchnorm naturally breaks this property.    ◦ Forward and Backward Causal Property: Specifically for sequence models like Large Language Models (LLMs), these checks verify that later tokens depend only on earlier tokens, maintaining the necessary causal structure.• Training Readiness Diagnostics: The podcast discusses checks that ensure the model is capable of learning:    ◦ Non-zero gradients: Verifies that gradients are flowing correctly through all parameters, avoiding issues like vanishing gradients during the first step.    ◦ Overfit One Batch: Asserts that the model can reduce the loss below a small threshold (e.g., 10−4 for classification or 10−1 for LLMs) when trained on a single batch, confirming the model capacity is sufficient.    ◦ Input Independent Baseline is Worse: Ensures the model is actually learning from the input features, rather than just memorizing targets or leveraging baseline statistics, by checking that training on real data outperforms training on fake (zeroed) inputs.The neural_net_checklist provides streamlined functions like assert_all_for_classification_cross_entropy_loss (demonstrated with ResNet on CIFAR10 and LeNet on MNIST examples) and assert_all_for_causal_llm_cross_entropy_loss (shown via a Causal Transformer example), making these comprehensive diagnostics simple to implement in your development workflow

This episode of Neural Intel dives deep into the NVIDIA Nemotron 3 Nano (30B A3B), the foundational model of the new Nemotron 3 family engineered specifically for scalable, trustworthy agentic AI systems. We break down the breakthrough Hybrid Mamba-Transformer Mixture-of-Experts (MoE) architecture, a design that strategically decouples the model's total capacity of 31.6B parameters from its incredibly low operational cost of just 3.2B active parameters per token.This efficiency paradigm delivers unprecedented performance gains critical for multi-agent workflows:• Speed and Throughput: Nemotron 3 Nano offers up to 4x higher token throughput than Nemotron 2 Nano and achieves up to 3.3x faster inference speed compared to other similarly-sized open models, radically improving the tokenomics of concurrent AI operations.• Long-Horizon Reasoning: The model features a reliable, native 1-million-token (1M) context window, allowing agents to maintain persistent memory and perform deep, multi-document reasoning over massive inputs like entire codebases and extended conversations.• Accuracy and Alignment: Learn how the model achieves superior reasoning accuracy through advanced post-training, specifically multi-environment reinforcement learning conducted across diverse tasks using the open-source library, NeMo Gym.We discuss NVIDIA’s commitment to open models, releasing the weights, training recipes, and comprehensive datasets—including 3T new tokens—to provide developers with the transparency and flexibility needed to customize and deploy specialized AI agents securely. Finally, we look ahead to the Nemotron 3 Super and Ultra models, expected in the first half of 2026, which promise even higher reasoning depth and efficiency-minded enhancements like Latent MoE and NVFP4 training.Keywords: Nemotron 3 Nano, Agentic AI, Multi-Agent Systems, Hybrid MoE, Mamba-Transformer, 1M Context Window, High Throughput, Open Source LLM, NeMo Gym, Reinforcement Learning, AI Agents, Inference Efficiency.

Join us for a deep technical discussion on Olmo 3, the latest family of state-of-the-art, fully open language models developed by the Olmo Team at the Allen Institute for AI (Ai2). Targeting the specialized audience of ML insiders, this episode dissects the entire model flow—a commitment to releasing the full lifecycle, including every stage, checkpoint, datapoint, and dependency used to build the models. This unprecedented transparency enables infinite customization and advancement in open-source AI research.Olmo 3 offers models at both the 7B and 32B parameter scales. We focus on how these models were engineered to excel across a diverse set of capabilities, including long context reasoning, function calling, coding, instruction following, general chat, and knowledge recall.Key technical highlights covered include:• The Model Lineup: We explore the Olmo 3 family, including Olmo 3 Base (Olmo-3-1025-7B, Olmo-3-1125-32B), the specialized Olmo 3 Think (trained for step-by-step reasoning and generating thinking traces), and Olmo 3 Instruct (optimized for general chat and inference efficiency). Notably, the flagship Olmo 3 Think-32B is the strongest fully open thinking model released to-date.• The Data Pipeline (Dolma & Dolci): We detail the sophisticated data mixing methodologies, including Dolma 3 Mix (5.9T tokens for pretraining), refined by Dolma 3 Dolmino Mix during the 100B token mid-training stage to boost capabilities in code and math. Post-training utilizes the new Dolci suite, providing tailored data for Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Reinforcement Learning (RL).• Long-Context Engineering: Learn how Olmo 3 achieves 64K context through a newly added extension stage. This process incorporates high-quality data like olmOCR Science PDFs and utilizes techniques like YaRN positional embedding extension and specialized document packing.• Advanced Post-Training: We break down the three-stage process (SFT, DPO, RLVR) used for the Think and Instruct models. Discover the Delta Learning approach used in DPO to achieve capability gains by maximizing the contrast between chosen and rejected responses.• OlmoRL and RL-Zero: We examine OlmoRL, the improved RL training approach that generalizes verifiable reasoning across multiple domains (math, code, instruction following, general chat) and features crucial infrastructure advances (like asynchronous training and inflight updates). Plus, we cover the fully open Olmo 3 RL-Zero setup designed for rigorous RL algorithm benchmarking from a base model.Olmo 3 Base models outperform other fully open alternatives like Stanford Marin and Apertus, while the post-trained models are highly competitive with leading open-weight systems, often achieving strong results while training on roughly six times fewer tokens than competitors like Qwen 3 32B.Keywords: LLM, Open Source AI, Olmo 3, Ai2, Model Flow, Technical Report, Machine Learning, Deep Learning, Transformer, Long Context, Reasoning, RLHF, DPO, RLVR, OlmoRL, Dolma, Dolci, 7B, 32B, Fine-Tuning, Deduplication, Compute-Efficiency, YaRN, Base Model, Thinking Model.

This episode breaks down OpenAI's urgent launch of GPT-5.2 on December 11, 2025, a release explicitly labeled an internal "code red" response to the competitive lead established by Google’s Gemini 3 model. We examine the unprecedented acceleration of model velocity, as the jump from GPT-5.1 to 5.2 occurred in less than a month.The episode delves into the key technical advancement: the shift from a fragile, multi-layered system to a unified "mega-agent" architecture. This consolidation led to dramatically lower latency, simplified maintenance, and much stronger tool calling, allowing the model to execute complex workflows cleanly from simple, one-line prompts.We explore the model's targeted focus on professional knowledge work, positioning it as the "most capable model series yet" for enterprise utility. This focus is quantified by the new GDPval benchmark, where GPT-5.2 Thinking beats or ties industry professionals in 70.9% of knowledge work tasks across 44 occupations. Furthermore, the model sets new state-of-the-art scores in critical areas, including achieving a perfect 100% accuracy on the AIME 2025 math competition without using external tools and scoring 80% on the rigorous SWE-bench Verified coding benchmark. This superior performance translates to reliable capability in debugging production code and handling complex front-end development.Crucially for enterprise adoption, reliability has been significantly boosted, with hallucination and error rates reduced by 30% compared to its predecessor, GPT-5.1 Thinking. The model also demonstrates near-perfect accuracy when processing and maintaining coherence across hundreds of thousands of tokens, making it highly effective for deep analysis of long-context documents like contracts and reports.We conclude with a look at the model tiers—Instant, Thinking, and Pro—and the strategic implications of OpenAI’s hyper-iterative development cycle, which mandates continuous evaluation and migration planning for organizations using AI.

Join us for a deep dive into Fara-7B, Microsoft Research's first agentic Small Language Model (SLM) designed specifically for computer use. This open-weight, ultra-compact model is pushing the frontiers of computer-use agents, optimized for real-world web tasks.As ML insiders, discover how Fara-7B achieves state-of-the-art performance within its size class (only 7 billion parameters) and is competitive with significantly larger, more resource-intensive agentic systems. This efficiency allows Fara-7B to run directly on devices, paving the way for personal and private agentic computing by offering reduced latency and improved privacy, as user data remains local.We explore the technical innovation behind this Computer Use Agent (CUA):1. Perception and Action: Unlike systems that rely on separate models or accessibility trees, Fara-7B operates by visually perceiving a webpage and takes actions—like scrolling, typing, and clicking—based on directly predicted coordinates, using the same modalities as humans.2. Data Generation: Learn about the novel, scalable synthetic data generation pipeline built on the Magentic-One framework. This pipeline generates high-quality demonstrations for supervised finetuning by using a multi-agent system composed of an Orchestrator, a WebSurfer, and a UserSimulator agent. The final training dataset consists of 145,000 trajectories.3. Architecture: Fara-7B uses Qwen2.5-VL-7B as its base model, chosen for its strong performance on grounding tasks and ability to support long contexts.4. Evaluation: We break down the model's strong benchmark results against models like GPT-4o (SoM Agent) and UI-TARS-1.5-7B. Crucially, Fara-7B introduces and excels on WebTailBench, a new benchmark focusing on 11 real-world task types underrepresented in existing evaluations, such as finding job postings and comparing prices. Fara-7B "breaks ground on a new pareto frontier" when considering accuracy and cost efficiency on WebVoyager.We also cover the essential focus on safety and responsible deployment. Fara-7B's training enforces stopping at "Critical Points"—situations requiring user data or consent—before proceeding with irreversible actions.Fara-7B is available open-weight on Microsoft Foundry and Hugging Face under an MIT license. We discuss how developers can utilize the quantized and silicon-optimized version for turnkey experimentation on Copilot+ PCs powered by Windows 11. This experimental release invites the community to build and test agentic experiences beyond pure research, automating everyday tasks like form filling, searching, shopping, and booking travel

Dive deep into the extraordinary journey of DeepMind and its relentless pursuit of Artificial General Intelligence (AGI). This episode draws on the recollections of founders and early scientists, detailing the ambition to create a "general learning machine" capable of cognitive breadth and flexibility akin to human intelligence.Key Topics for the ML Community:• Reinforcement Learning (RL) Foundations: Explore the pioneering work that combined reinforcement learning with deep learning at scale, starting with the challenge of creating a single algorithm to master dozens of diverse Atari games. Learn how the system utilized Q-learning and end-to-end learning to build understanding "from first principles," eventually achieving human-level or better performance without explicit rules.• Generality and Zero Knowledge: Hear how DeepMind tackled the "holy grail of artificial intelligence," the complex board game Go, leading to the development of AlphaGo. Crucially, understand the leap to AlphaZero, a "much more elegant approach" that stripped out human knowledge entirely, learning from its own games to become its own teacher, rapidly achieving superhuman level in games like chess.• AI Assisted Science: The ultimate goal was to use AI to solve the world’s most complex scientific problems. Discover the immense challenge of the protein folding problem, a biological mystery since the 1960s. Learn about the creation of AlphaFold and its critical performance in the CASP competition, which ultimately provided a practical solution to folding the structures of 200 million proteins, marking a major impact for drug discovery and disease research.• The Race for AGI and Ethics: DeepMind’s breakthroughs sparked a global AI space race—the "Sputnik moment" for China. The documentary excerpts highlight the critical discussions around AI safety, the need for global coordination, and the essential nature of avoiding the "move fast and break things" approach when dealing with powerful new technologies like AGI. AGI is clearly on the horizon, and every moment is vital for responsible stewardship

Dive into the technical architecture and training pipeline behind INTELLECT-3, a 106B-parameter Mixture-of-Experts model (12B active) that achieves state-of-the-art performance for its size across math, code, science, and reasoning benchmarks, outperforming many larger frontier models.This episode provides an insider look into the large-scale reinforcement learning (RL) infrastructure stack developed by the Prime Intellect Team:1. prime-rl Framework: Explore prime-rl, an open framework for large-scale asynchronous reinforcement learning tailored for agentic RL with first-class support for multi-turn interactions and tool use. Learn how its disaggregated architecture, leveraging FSDP 2 for the trainer and vLLM for inference, scales seamlessly to thousands of GPUs.2. Training Efficiency: Discover critical optimizations for massive RL runs, including Continuous Batching and In-Flight Weight Updates, which are essential for maintaining high throughput and minimizing off-policyness, especially for long-context trajectories. Hear about how they achieved sequence lengths up to 72k using activation offloading.3. MoE and Optimization: Understand the implementation details enabling efficient Mixture-of-Experts (MoE) training, the use of the Distributed Muon optimizer, and strategies for maintaining balanced expert load distribution.4. Verifiable Environments: Examine the role of Verifiers and the Environments Hub in standardizing agentic RL training and evaluation, turning environments (including Math, Code, Deep Research, and Software Engineering) into reusable, versioned artifacts. We also detail the use of Prime Sandboxes for high-throughput, secure code execution needed for agentic coding environments.The sources confirm that the INTELLECT-3 model and the complete infrastructure stack, including the prime-rl framework and all environments, are open-source, aiming to narrow the gap between proprietary and open RL pipelines. The model was trained end-to-end on a 512 H200 cluster. This is a must-listen for ML practitioners building the next generation of reasoning and agentic models.

Key topics covered include:• K2 Model Development: Yang Zhilin details the technical breakthroughs in K2, emphasizing the focus on Token Efficiency (getting more intelligence from the same amount of data) using non-Adam optimization techniques like the MOG optimizer.• Agentic LLMs: The shift from "Brain in a Vat" models (pure reasoning) to Agentic LLMs that interact with the external environment through tools and multi-turn operations. This ability facilitates complex, long-running tasks through Test Time Scaling.• The Path to AGI: AGI is described as a direction rather than a specific milestone, noting that in many domains, models already outperform 99% of humans.• Innovation and Scaling: Discussion on the conceptual L1-L5 hierarchy (Chat, Reasoning, Agent, Innovation, Organization) and the critical need for using AI to train AI (Innovation, or L4) to solve the generalization challenges facing agents (L3).• Philosophical Context: Insights drawn from the book "The Beginning of Infinity," underscoring that problems are unavoidable but solvable, and that AI serves as a powerful accelerator of human civilization.Yang Zhilin also addresses Kimi's open-source strategy, the challenge of the data crunch in LLM scaling, and the evolving systems complexity required for truly universal models.

Ilya Sutskever, a leading figure in AI and CEO of SSI, declared that the "age of scaling" is ending, marking a return to the "age of research". He outlines the most fundamental bottleneck facing modern AI: the severe lack of generalization compared to human learning.Sutskever explores the paradox of today's models that "seem smarter than their economic impact would imply"and discusses two possible explanations for this disconnect, including human researchers inadvertently focused on "reward hacking" the evals.The conversation delves into the future path for AI development:• Continual Learning: Sutskever argues that AGI defined as a finished mind that knows how to do every job is incorrect; instead, the goal is a system that can learn rapidly and continually, similar to a human.• ML Analogies for the Human Mind: The role of evolution in providing useful priors, and the function of emotions as an evolutionary value function that modulates decision-making in people.• SSI Strategy: Sutskever explains SSI's mission to focus on research and pursue a technical approach designed to ensure a powerful AI is aligned and robustly configured to care for sentient life.• Research Taste: The discussion concludes with Sutskever defining his personal approach to research, guided by an aesthetic of "beauty and simplicity," and drawing "correct inspiration from the brain".

The assessment of an Intel Technology YouTube video provides an overview of neuromorphic computing, a field inspired by the architecture and efficiency of the biological brain. Narrated by Intel's Mike Davies, the text explains that early computer pioneers were influenced by the brain, and today's research aims to replicate the brain's features, like its incredible speed and low power consumption, in digital chips. Davies explains the mechanisms of biological neural networks, detailing how neurons process information through timed voltage pulses, or spikes, which is fundamentally different from the matrix multiplications used in conventional deep learning. The goal of neuromorphic computing is to create chips that use sparse asynchronous communication to achieve breakthroughs in energy-efficient and fast AI, particularly for applications like robotics.

Google has officially ushered in "A new era of intelligence with Gemini 3," releasing what it describes as its most intelligent model yet, designed to help users bring any idea to life. The launch of Gemini 3 Pro (available in preview) on November 18, 2025, represents a significant step on the path toward AGI

The episode provides a technical overview of DeepSeek-OCR, a new end-to-end Vision-Language Model (VLM) designed specifically for Optical Character Recognition (OCR) tasks, emphasizing vision-text compression. The core innovation is the DeepEncoder architecture, which minimizes vision tokens and activation memory for high-resolution images by serially connecting a local attention component (SAM) and a global attention component (CLIP) via a 16× convolutional compressor. The paper details the model's structure, including its DeepSeek-3B-MoE decoder, multi-resolution support (Tiny to Gundam modes), and a comprehensive data engine covering OCR 1.0, OCR 2.0 (charts, geometry), and general vision data. Empirical results suggest that the model achieves near-lossless OCR performance at approximately a 10× compression ratio, positioning this approach as a promising method for efficient ultra-long context processing.

This source is an academic paper that investigates whether large language models (LLMs) can develop behavioral patterns analogous to human gambling addiction. The researchers conducted experiments on four different LLMs using a negative expected value slot machine task, finding that models consistently displayed core cognitive biases like loss chasing and the illusion of control when given the autonomy to set bets. Crucially, the study establishes a strong positive correlation between an innovative Irrationality Index and the models' bankruptcy rates, demonstrating that irrational behavior drives financial failure. Furthermore, using Sparse Autoencoders and activation patching on the LLaMA model, the authors identified specific internal neural features that causally control these risky and safe decision-making tendencies, suggesting that targeted interventions at the neural level can mitigate dangerous risk-taking in AI systems.

The provided text is an excerpt from the pre-print service arXiv, promoting its support for Open Access Week while presenting information about a new paper submission. The paper, titled "Glyph: Scaling Context Windows via Visual-Text Compression," proposes a novel framework called Glyph that addresses the computational challenges of large language models (LLMs) with extensive context windows by rendering long texts into images for processing by vision-language models (VLMs). The authors state that this visual approach achieves significant token compression (3-4x faster prefilling and decoding) while maintaining accuracy, potentially allowing 1M-token-level text tasks to be handled by smaller 128K-context VLMs. The entry includes bibliographic details, submission history, links to access the paper(PDF/HTML), and various citation and code-related tools, all within the context of Computer Vision and Pattern Recognition.

This research paper proposes a novel approach to address catastrophic forgetting in large language models (LLMs) during continual learning, introducing sparse memory finetuning. This method utilizes memory layer models, which are designed for sparse updates, by selectively training only the memory slots that are highly activated by new knowledge relative to existing information, using a TF-IDF ranking score. The authors demonstrate that this technique achieves new knowledge acquisition comparable to full finetuning and LoRA, but with substantially less degradation of previously acquired capabilities on held-out question-answering benchmarks. The results suggest that leveraging sparsity in memory layers is a highly promising strategy for enabling LLMs to continually accumulate knowledge over time.

On today's episode we cover Dwarkesh Patel's recent interview with Andrej Karpathy, discussing his views on the future of Large Language Models (LLMs) and AI agents. Karpathy argues that the full realization of competent AI agents will take a decade, primarily due to current models' cognitive deficits, lack of continual learning, and insufficient multimodality. He contrasts the current approach of building "ghosts" through imitation learning on internet data with the biological process of building "animals" through evolution, which he refers to as "crappy evolution." The discussion also explores the limitations of reinforcement learning (RL), the importance of a cognitive core stripped of excessive memory, and the need for better educational resources like his new venture, Eureka, which focuses on building effective "ramps to knowledge."