DiscoverTraining Data
Training Data
Claim Ownership

Training Data

Author: Sequoia Capital

Subscribed: 295Played: 5,694
Share

Description

Join us as we train our neural nets on the theme of the century: AI. Sonya Huang, Pat Grady and more Sequoia Capital partners host conversations with leading AI builders and researchers to ask critical questions and develop a deeper understanding of the evolving technologies—and their implications for technology, business and society.


The content of this podcast does not constitute investment advice, an offer to provide investment advisory services, or an offer to sell or solicitation of an offer to buy an interest in any investment fund.

114 Episodes
Reverse
Today’s agents are stuck in what Ronak Malde calls “pseudo continual learning”: intelligent, but updated in offline batches, like a paper road map you can’t reroute mid-trip. Ronak, co-founder of Trajectory, explains why closing that loop is the next learning paradigm, and why training your own model, which used to take a dedicated research team, months of work and hundreds of millions in compute, is now something a company can do in a few clicks with their own data. He details two research bets they are making available to everyone with their product: 1) a variant of GRPO that removes the need for a simulated environment by learning from single real-world rollouts instead of 16 replays of the same task, and 1) a meta-grader that increased data efficiency by 100x. Ronak explains how Harvey trained a model six times cheaper than frontier, improving an open-source baseline by over 4x on Harvey's own legal benchmark. He defines the difference between continual learning and recursive self-improvement and predicts product builders will become gardeners, setting trellises for products that grow themselves. Hosted by Pat Grady and Lauren Reeder, Sequoia Capital 0:00 – Introduction 0:45 – Pseudo continual learning: paper maps vs Google Maps 2:48 – Do we need new architectures? Rich Sutton, David Silver, and the missing layer 4:33 – From a $100M consulting model to a product: lessons from SWE-1 at Windsurf 7:36 – Harvey: 6x cheaper than frontier, 4% to 18% on its own benchmark 8:55 – Why "Trajectory": the unit of work is a task, not a token 10:13 – The primitives of continual learning: traces, replayable environments, reward 12:56 – Group Size 1 and the meta-grader: training without an environment or a labeled reward 16:16 – The platform: train, eval and deploy in one click, and who it's for 18:58 – Rogo and intelligence density 20:14 – What's next: curiosity, and why continual learning beats RSI on alignment 24:19 – Research and product under one roof, and the product builder as gardener 27:26 – Why work at Trajectory, and the optimistic case: from 1% to 99%
Two years after its founding, Stark has roughly 700 employees and strike drones in the field in Ukraine. CEO Uwe Horstmann and CTO Johannes Schaback argue that the real lesson of that war is not any single weapon: the innovation ecosystem behind it is the wonder weapon. They explain why Stark designs every system for mass manufacturing from day one, runs 10 to 15 programs like a venture portfolio, and views the staff sergeant who operates a system as just as important as the general who buys it. Johannes recounts the publicly reported failed trial that became a forging moment, as engineers worked around the clock at a freezing test range until they landed three consecutive hits. They discuss NATO’s principles of responsible AI use and the hard-coded constraints in Stark’s autonomy stack. And Uwe lays out his next big idea: defense contracts structured like cloud contracts with surge capacity built in, so production itself becomes deterrence. Hosted by Pat Grady and Luciana Lixandru, Sequoia Capital Chapters 0:00 – Introduction 1:03 – Billions in contracts in 18 months: Stark's four-part playbook 2:28 – The staff sergeant matters as much as the general, and wood beats plastic 7:42 – Why Europe is ready for a neo prime, and why procurement reform will last 11:45 – Startup DNA, venture-style programs, and "as complex as a washing machine" 16:27 – Born out of fire: the failed trial and the round-the-clock comeback 21:13 – Where Stark finds talent, and why defense is no longer off-limits 23:48 – Anatomy of a strike, from the wooden crate to the kill zone 27:22 – Vertical integration, multiple domains, and shifting left on manufacturing 32:53 – After the Hugging Face hack: guardrails on autonomous weapons 35:56 – Matching the primes on R&D and scaling to 700 people in two years 39:24 – Deterrence by contract, and Europe's next chapter with the US
Robotics never got the internet's gift of free training data, so someone has to manufacture it. Josh Gao, founder and CEO of Mecka AI, runs one of the largest robot data factories, and he explains what that actually involves: custom baseball caps and tactile gloves because nothing off the shelf captures force, 100-gigabit upload lines to move hundreds of petabytes, and custom quality gates. Josh lays out the training recipe the industry is converging on (roughly a million hours of egocentric human data, tens of thousands of hours of robot data, and a few hours of in-domain teleop ) and why tactile and simulation data still have to enter the "data pot of soup." He argues the field is still orders of magnitude behind Tesla's FSD, which spent a decade on two degrees of freedom while a robot hand has dozens. His take: there won't be a single GPT-3 moment for robotics, but a steady unlock of commercial tasks, starting with constrained industrial work and ending, he hopes, with robots mining asteroids. Hosted by Sonya Huang, Sequoia Capital 0:00 – Introduction 1:46 – What Mecka is: pixels, actions and logic 2:45 – Is robot data economically feasible without the internet? 4:15 – Teaching robots about carbon life 5:27 – Cloud kitchens, stablecoins and the path to robotics 7:03 – Entering robotics with a fresh lens 8:45 – What egocentric data is and why Mecka bet on it 10:09 – The first pilot: aunties and uncles with a mobile app 10:42 – Force data and tactile gloves 11:55 – Inside the operations of a data factory 14:41 – Heat-mapping the taxonomy of human work 15:42 – Quality gates, fraud and the McDonald’s POV videos 17:44 – Earning a margin as a data intermediary 19:33 – Millions of hours vs. Tesla’s decade 20:55 – Why it isn’t binary: unlocking commercial tasks along the way 22:02 – Home data is plentiful; home robots are the hardest 23:24 – Useful robots can be deployed today 24:35 – When robotics gets an FSD-style flywheel 26:42 – Deployment diffusion and the modern systems integrator 28:21 – Favorite vertical robot categories; mining an asteroid 30:08 – The hardware slate: Waymo-fying people 30:55 – EgoVerse 32:04 – The data pot of soup: the converged training mix 33:54 – Move 37 for robotics: AlphaGo and Federer 36:02 – World models and the training mix 36:58 – Will scaling the current mix get us to the promised land? 39:05 – Will the AGI labs steamroll robotics? 40:50 – Sci-fi roots and the Kawasaki Corleo 42:44 – How many robots in a decade?
David Silver spent two years building AlphaGo on human data. When he removed human data altogether building AlphaGo Zero, the AI surpassed AlphaGo in under two days. He left DeepMind to run that same bet in pursuit of superintelligence. David and Lasse Espeholt, co-founders of Ineffable Intelligence, explain their plan to build a “superlearner”—an agent that starts knowing nothing and discovers science, mathematics, and language for itself by direct experience instead of through a pre-training run. Silver argues that human-language reasoning is a ceiling, not a foundation, that alignment via chain-of-thought monitoring is already reaching its limits, and that the science of how superintelligences coexist should be worked out in simulation before anything touches the real world. Espeholt describes the GPU cluster and the lineage from IMPALA and SEED RL that makes the experiment possible. Silver’s forecast: strong evidence within a year, superintelligence in three. Hosted by Sonya Huang, Sequoia Capital 0:00 – Introduction 1:09 – What held true from Go to today, and what the LLM era changed 3:35 – Why leave DeepMind to start a new lab 5:58 – Why "Ineffable" (and the secret second reason) 7:03 – Lasse's path: scaling RL at DeepMind and Google Brain 9:31 – Defining a super learner and first contact 12:35 – Super learners vs. LLMs that make discoveries 13:59 – Day to day: the RL stack, crucibles and human-journey milestones 16:13 – Don't breadcrumb the objective: simple rewards, emergent complexity 21:16 – What a crucible looks like 24:14 – Go: one agent is boring, two is infinite 25:41 – Will agents rediscover language? 27:22 – Removing human priors: AlphaGo to AlphaZero 32:05 – Keeping human bias out of the crucibles 33:35 – Confidence, timing and super forecasting 36:24 – Is compute the driver? 38:13 – The ceiling on simulation and Rich Sutton's objection 43:44 – Alignment and the science of superintelligence 47:45 – Monitoring agents that don't think in English 51:01 – Pacing the frontier 52:54 – The compute setup: Vera Rubin on GCP 55:18 – Training, inference, simulation and RL scaling laws 57:40 – IMPALA and SEED RL 1:00:58 – 20 people and the compute-per-person ratio 1:04:49 – A research-led culture 1:05:47 – Predictions
At 100,000-accelerator scale, something fails multiple times an hour, which is why Google's Chief Technologist for AI Infrastructure Amin Vahdat thinks FLOPS is a vanity metric. The metric that matters is what he calls goodput: the useful work a workload actually delivers through real failures. Amin walks through the calculus that split the TPU line into 8i and 8t for the first time, why the TPU's core primitives haven't changed since v1, and how Google and DeepMind co-design in the same rooms, intercepting chip architectures mid-flight before tape-out. He explains why long-horizon agents are sending demand for CPUs and storage through the roof alongside accelerators, how optical circuit switches reroute light to a spare rack in milliseconds, and why Google would rather wait on a utility than build its own gigawatt. We also cover orbital data centers and the multi-megawatt rack of 2036. Hosted by Sonya Huang, Sequoia Capital 0:00 – Introduction 1:47 – What makes a data center an AI data center 5:30 – Goodput, not FLOPS: holding yourself accountable when something fails every hour 11:52 – Doubling token capacity every six months, and where the gains actually come from 15:32 – The TPU bet: from a contrarian call in 2013 to splitting 8i and 8t 23:30 – The case for and against co-design 26:11 – Shoulder to shoulder with DeepMind: intercepting chips mid-flight 34:16 – Long-horizon agents change the shape of the data center 37:50 – Optical circuit switching and the state of networking 42:35 – Power is the binding constraint: utilities, gigawatts, and sizing a data center 49:23 – Training vs. serving clusters, seven-year-old TPUs, and open standards 58:29 – Orbital data centers and the supercomputer of 2036
loading
Comments