Humans On The Loop: Wisdom for an Age of Magical Technologies

With AI on everyone’s mind, we have to ask: what does it mean to think?When does the brain actually create predictive models and rely on mental representations — and when is it really just a matter of inputs and outputs with no simulation going on in between?Why do we feel pain and find things funny? Not just avoid damage or understand the logic of a joke, but actually feel these things?Agreeing on what we don’t know matters, not least because trillions of dollars are invested in the dream of artificial general intelligence — but if thinking emerges in AI, how would we even know where to look? Behavioral indicators aren’t enough. What is the internal structure that would let us say with confidence “There’s real thinking going on here?”Because as the success of deep neural networks makes clear, most intelligence doesn’t need this extra layer. Most intelligence isn’t thinking — at least, at the level of the whole organism or AI system. And, as much as we want to believe otherwise, maybe none of it has to be.We need to remind ourselves what we don’t know. This is why guest co-host Adam Safron and I called on neurologist and neuroscientist John W. Krakauer (Website | X | Google Scholar), Director of the Johns Hopkins Brain, Learning, Animation, and Movement Lab and External Professor at the Santa Fe Institute, who has spent decades studying the modularity of the brain and the relationship between sensorimotor functions and deliberative thought through research on stroke patients.Our conversation with John gets at the very heart of the biggest taboo in cognitive science: that there is a difference between intelligence and thinking…that we’re conscious, and we don’t know why, and these distinctions matter to intelligence research. As AI demonstrates more and more capability to do things we thought it took conscious thought to achieve but shows no indisputable concrete evidence of being conscious, it turns out science still doesn’t understand what consciousness is.Upcoming EventsConsider becoming a patron to participate in our upcoming salons, including:• November 20 — Accelerando by Charles Stross (notes & recording from Part 1 here)• December 16th — The Dimensions of Experience: A Natural History of Consciousness by Andrew P. SmithSupport Work The Newsfeed Algorithm IgnoresThanks to the Survival and Flourishing Fund for their support of this mini-series! I do not run ads, so if you get value from this work, you might like to:✨ Become a founding member to access unreleased episodes & online courses✨ Hire me as an advisor or public speaker✨ Back my research and creative projects with a tax-deductible donation✨ Browse and buy the books we mention on the show at Bookshop.org✨ Buy my art — paintings, prints, blankets (commissions open)✨ Learn more about how we’re applying these ideas at Atlas Research Group, my team building commons-owned tools for collective intelligenceAnd if you’re asking, “IN THIS ECONOMY?” but still want to show your support, tell a friend about the show and you’ll both have something new to talk about:Chapters00:00 Teaser01:46 Intro08:28 Meet John Krakauer10:53 Against Internal Models13:01 Replay and RL Critique16:37 Mental (Not Neural) Representation20:55 What Counts as Evidence24:03 Why Humans Deliberate26:56 AI Mirrors the Taboo28:42 Deliberation Taboo Framework32:04 Chain of Thought Confabulation34:30 Slow Science and Creativity38:15 Cognition vs Consciousness History43:57 Pain and Conscious Advantage46:09 Illusionism and Dennett49:12 Working Memory Bottleneck Debate51:03 Bennett On Simulation Limits53:19 Defining Real Emergence56:38 Inside LLM Representations57:56 NeuroAI Chick Experiments01:04:08 Efficiency And Evolutionary Pressure01:07:00 Laughter, Feelings, Deliberation01:13:53 Basal, Cortical, & Conscious Intelligence01:19:57 World Policies, Not Models01:23:27 Language’s Role & The Meaning Gap01:27:54 Tacit Learning & One-Shot Rules01:30:18 Admitting The Deliberation Taboo01:31:12 Explanatory Gaps And Coarse Graining01:37:28 ClosingMentioned ResourcesExplore the entire open-access special issue here.And whether you read the papers or not, be sure to check out this illuminating interactive discourse map by Van Bettauer of Ideoscopic to help you navigate where these researchers agree, disagree, and point toward future study:(Long-time fans will also want to check out his re-imagined interface for AskFutureFossils.com, complete with simulated debates between my guests!)Mental representation without neural representation: Understanding the evidenceJohn W. Krakauer, William RamseyThe Deliberation TabooJake Quilty-Dunn, John W. KrakauerLarge language models and emergence: a complex systems perspectiveDavid C. Krakauer, Melanie Mitchell, John W. KrakauerArtificial intelligence for science: The easy and hard problemsRuairidh Battleday & Sam GershmanEmpowerment gain and causal model construction: children and adults are sensitive to controllability and variability in their causal interventionsEunice Yiu, Kelsey Allen, Shiry Ginosar, Alison GopnikA Brief History of IntelligenceMax BennettFrom Bacteria to Bach and BackDaniel DennettThe Rediscovery of the MindJohn SearleConcepts at the InterfaceNicholas SheaThe cultural transmission of tacit knowledgeHelena Miton, Simon DeDeoThis conversation was recorded before Anthropic announced their claim Claude does in fact form a “global workspace” and representational models — work that suggests to some researchers that frontier LLMs are in fact now possibly doing something that looks like actual thought. But is it? This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.humansontheloop.com/subscribe

For Part Two in our Continual Learning mini-series, Dr. Nadav Amir of The University of Ottowa joins me and guest co-host Dr. Adam Safron of Tufts University for a trialogue on the fundamental role of goals in intelligence.Nadav draws on behavioral neuroscience, Buddhist philosopher Dharmakirti, control and information theory, and John Vervaeke’s work on “relevance realization” to formally articulate what many wisdom traditions have argued for millennia: what you take to be “reality” — your world and self — is a function of goal-based representations, state descriptions and reward functions create each other, and “what is” and “what do I care about” are two sides of the same thing. You can’t separate what you want from what you experience and what you are capable of. But this means that if you change your goals your reality will shift, and your sense of self with it. The opposite is also true: change the way you organize the the ineffable structure of reality into a conceptual framework, and you’ll care about very different things. Thanks to the pace of change we’re living through this kind of transformative experience now, so it’s a good time to ask:If selves are a kind of falsification, why do we have them in the first place? Does AI need goals of its own in order to be truly intelligent?And perhaps the deepest question we ask in this episode:How if at all can we develop better frameworks for deciding what we want our intelligent machines to become — and who we want to become alongside them?Thanks to the Survival and Flourishing Fund for their support of this mini-series. If you’d like more of this kind of work in the world:✨ Support my research, writing, and podcast with a tax-deductible donation✨ Become a founding member to access my online courses, including Jurassic Worlding and How To Live In The Future✨ Hire me for public speaking or advisory work with your organization✨ Browse and buy the books we mention on the show at Bookshop.org✨ Stream and download my tunes at artist-owned music co-op Subvert.fm✨ Continual Learning theme song: “Substrate Dynamics” by Neon Chameleon✨ Learn more about how we’re applying these ideas at Atlas Research Group, my team building sovereign infrastructure for collective intelligenceChapters0:00:00 Teaser0:01:48 Intro0:05:42 Who is Nadav Amir?0:08:15 Why we can’t separate goals from descriptions0:15:32 The Ugly Duckling Theorem & why taxonomy is not reality0:25:07 Evolution, karma, and non-conceptual awareness0:31:04 If selfhood is wrong, when is it practical?0:34:14 Suffering without a self & the value of wrong descriptions0:38:14 Non-conceptual experience & active inference0:41:07 Multiple nested hierarchical descriptions in minds & economies0:46:07 What’s missing from current approaches?0:50:09 Does greater agency mean better optimization or greater adaptability?0:57:04 Empowerment-based causal learning without a ground truth0:59:06 Intermission0:59:58 Transformative experiences & multiple levels of granularity1:05:19 No fixed models, no fixed environments1:10:55 Fundamental tradeoffs1:16:57 Reverse-engineering the centered self1:19:54 Does AI need goals to be intelligent and can we control AI if it has them?1:26:47 Does Nadav’s framework pertain for all possible minds?1:32:37 Mind as a property of collectives & goals as embedded in environments1:41:33 How can Nadav’s framework improve AI governance discourse?1:48:59 OutroMentioned ResourcesA Dharmakīrtian Model of Relevance Realization in Cognitive Agentsby Nadav Amir & John DunneAn exchange of letters on the role of noise in collective intelligenceby Daniel Kahneman et al.Cognitive glues are shared models of relative scarcities: the economics of collective intelligenceby Michael Levin & Benjamin LyonsReverse-Engineering The Centered Selfby L.A. Paul et al.Agency and Experience: Buddhist and Cognitive Perspectivesa Yin Cheng Conference @ PrincetonKnowing and Guessingby Satosi WatanabeOntological Laughter: Comedy as Experimental Possibility Spaceby Timothy MortonSelves as Perspectives: From Biological Life to Superintelligence and a Bodhisattva Projectby Thomas Doctor et al.What The Frog’s Eye Tells The Frog’s Brainby J. Y. Lettvin et al.The Center for the Study of Apparent SelvesExplore the entire open-access special issue here:And whether you read the papers or not, be sure to check out this illuminating interactive discourse map by Van Bettauer of Ideoscopic to help you navigate where these researchers agree, disagree, and point toward future study:(Long-time fans will also want to check out his re-imagined interface for AskFutureFossils.com, complete with simulated debates between my guests!) This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.humansontheloop.com/subscribe

What does it actually mean for a machine to “understand” the world? Are today’s auto-regressive LLMs truly reasoning, or is statistical text prediction fundamentally distinct from genuine deliberative agency and causal inference? And how do we move away from brittle, post-hoc safety patches toward intrinsic, system-level alignment?This week begins Continual Learning, a new mini-series co-hosted with my friend and colleague, cognitive scientist Dr. Adam Safron of the Allen Discovery Center at Tufts University and the Active Inference Institute. For the last year, we’ve been working together with the support of Survival and Flourishing Fund to advance scientific understanding and silo-crossing conversation around AI capabilities, alignment, and regulation—centered on a special issue of Philosophical Transactions of The Royal Society A on World models in natural and artificial intelligence co-edited by Adam and Michael Levin. The next several episodes are a meaningful detour into this work.Over the coming season, we’ll dive deep into cognitive neuroscience, complex systems science, the study of narratives, and Buddhist epistemology to explore what true world modeling entails. This episodes launches that investigation by identifying major major themes from the special issue and connecting dots between its papers. (Strap in, because we move a million miles an hour.) Some of the questions we raise include:* How do we rigorously define what a world model is—and isn’t?* Do machines need goals, intrinsic motivation, and deliberation to truly think?* What is the relationship between world-modeling and agency?* Where can we look for evidence of emergent structure in scaling LLMs, and what does it mean if we don’t find it?* How can we structure scientific collaboration to ask better questions about the future of human-machine co-evolution…and what might it take for machines to actively participate in that inquiry?Explore the entire open-access special issue here.And whether you read the papers or not, be sure to check out this illuminating interactive discourse map by Van Bettauer of Ideoscopic to help you navigate where these researchers agree, disagree, and point toward future study.(Long-time fans will also want to check out his re-imagined interface for AskFutureFossils.com, complete with simulated debates between my guests!)Subscribe for amazing conversations with Nadav Amir, John Krakauer, Fritz Breithaupt, Michael Levin, and many more!✨ Hire me to help you and your team identify your blind spots and make better sense✨ Support my research, writing, and community work with a tax-deductible donation✨ Become a founding member to access my online courses, including Jurassic Worlding and How To Live In The Future✨ Browse and buy the books we mention on the show at Bookshop.org✨ Stream and download my tunes at artist-owned music co-op Subvert.fm✨ Continual Learning theme song: “Substrate Dynamics” by Neon Chameleon✨ Learn more about how we’re applying these ideas at Atlas Research Group, my team building sovereign infrastructure for social coherence and collective intelligenceChapters00:00 Teaser01:58 Intro06:04 We’re finally doing this06:35 Adam’s context on this series11:38 Michael’s context on this series16:09 Bringing together this special issue20:34 The cognitive equivalent of an airplane wing?26:56 Connecting goals and intrinsic motivation31:29 Requisite diversity in scientific and machine intelligences38:41 Is there an “I” in AI, and does it need constraints?41:48 Does consciousness emerge, and how is it not capability?47:45 Misattributing mind vs. missing mind55:28 Why is it so hard to make an AI scientist?1:02:43 The promise of a new economy1:08:52 Embodiment and alternative architectures1:13:45 Where do we go from here?1:17:34 OutroOther MentionsActive Inference Symposium 2025 roundtable discussion with Karl J Friston, Michael Garfield, Adam Safron, Alexander Ororbia, Hongju Pae:https://www.youtube.com/watch?v=VTivWre-8KkRay Kurzweil — The Singularity is NearErik Hoel — The Overfitted Brain: Dreams Evolved to Assist GeneralizationW. Brian Arthur — The Nature of TechnologyNicholas Carr — The Glass CageKevin Kelly — The Handoff to BotsSteven Johnson — Revenge of The HumanitiesHumans On The Loop is committed to better collective sensemaking and the production of public goods. Thank you for your support! Learn more here. This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.humansontheloop.com/subscribe

Today’s guest is William Morgan, the co-founder of Restless Egg, a London-based incubator and accelerator designed specifically for a new class of “artist-founders” whose work synthesizes art, technology, and commercial product design.His thesis: The tech sector’s obsession with optimization can’t help us discover the genuinely novel forms of human experience that this intelligent substrate makes possible. If we want something truly better, we need to cultivate the taste and discernment to know which futures are worth living in — and radically expand our imagination regarding what human-machine relationships can actually look like.✨ Become a founding member to access my online courses, including Jurassic Worlding and How To Live In The Future✨ Browse and buy the books we mention on the show at Bookshop.org✨ Stream and download my music at artist-owned (!) Subvert.fm✨ Learn about Atlas Research Group, my new team building sovereign infrastructure for social coherence and collective intelligence.Chapters00:00 Teaser: Taste & Sovereign Choice02:46 Intro 06:41 Meet William Morgan 08:05 The Origins of The Avant-Garde 10:09 Taste and Abundance 12:12 Luxury Tech and Experience 15:57 Why Experiment? 20:21 Venture Models and Niches 25:20 The Sense Organs of Society 30:51 Rethinking Venture Incentives 38:54 Slop and Subculture Rebellion 47:13 Beyond “In” or “Out” 50:51 Founder Experiments Showcase 57:03 Scenius is Real ValueMentionedRestless Egg’s Half Dozen NewsletterArt Is Everything You Don’t Have To Do by Cory DoctorowThe Usefulness of Useless Knowledge by Abraham FlexnerThis collective is radically rethinking what it means to make art by Thom Waite at Dazed DigitalThe Dimensions of Experience by Andrew P. SmithMount Analogue  by René DaumalThe Nerves of Government  by Karl DeutschWhy Software is Eating The World by Marc AndreessenY Combinator’s Simple Agreement for Future EquityCommon As Airby Lewis HydeStanding by Wordsby Wendell BerryRight Story, Wrong Storyby Tyson YunkaportaFall by Neal StephensonThe Politics of Visionby Lydia NochlinThe Key to Science Fictionby Damien WalterFlora WeilAntithykeraCCRULuciana ParisiGeorge BatailleMatthew David Segall This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.humansontheloop.com/subscribe

✨ Become a founding member to access my online courses, including Jurassic Worlding and How To Live In The Future✨ Browse and buy the books we mention on the show at Bookshop.org✨ Stream and download my music at artist-owned (!) Subvert.fm✨ Learn about Atlas Research Group, my new team building sovereign infrastructure for social coherence and collective intelligenceIn the last episode, my conversation with C. Thi Nguyen explored “value capture”: his term for what happens when our scoring systems define what we care about and ultimately our identities. In this episode, we ask what it means to reverse this process — what you might call “value emancipation” — with Andrea Farias (LinkedIn), a Barcelona-based researcher and builder whose guiding mission to is to support the transition to a regenerative civilization aligned with the flows of our planet.Andrea investigates how digital technologies can accelerate this transition, crafting information ecosystems that reimagine knowledge creation and resource allocation. But her path to this life required some serious unlearning and a rocky road out of her prior incarnation as a healthcare tech product strategist.We invite you to ask the same questions she did, and which we ask in this converstion:• When I decouple from the desires I’ve identified with, what is worth taking their place?• How do we design and adopt technologies from this new, more spacious identity?• How does the local determination of plural value help us restore necessary context to human-scale decision-making?• How do we navigate the tensions between the place-ful realm of community and the placeless realm of global coordination?• Where do we need friction in our digital lives, and how can ecology and bioregionalism inspire visions for a better Web?• What does it mean to be “local to an idea or a narrative” and how does that cyber-locality interface with geographic locality?• Where do we still want abstraction for coordination at scale?Tune in for a deep dialogue on how to care for the processes that actually create life — and what it means to enact regenerative principles to personal and collective health, technology and economy.(Fun fact: although she wasn’t at the time of this recording, Andrea is now a member of Atlas Research Group! So we will definitely be doing more together…)Special AnnouncementJoin me at the Weirdosphere online learning platform for “Transcendence in the Age of AI” — where we’re hosting an interactive screening of Steven Spielberg’s A.I. (2001) on June 23rd and chasing it on June 25th with a deep-cut conversation between myself and two wonderful writers and film-makers: Weird Studies co-host JF Martel and Joel Gunz of Macguffin Media. Fresh ideas guaranteed! The viewing party is free to all; the follow-up dialogue and group discussion is $20 USD.Register here. Founding Members on Substack and Patreon can join for free, as always! Reach out if you are one and would like the free registration link.Chapters00:00 Replacing Habits With Values01:47 Introduction05:39 Job Creation vs. Job Destruction08:04 Enoughness & Bigger Desires Than “More”11:00 Andrea’s Story of Crisis & Transformation23:04 Limits, Care, & Post-Growth30:44 Bioregionalism, Currency, And Web340:40 Tokenization Tradeoffs42:09 Governance Starts Local44:42 Rewilding Digital Biomes49:42 The Fractal Cozy Web55:29 AI Translation And Legibility01:04:48 Bioregional Finance Experiments01:11:16 Protocols, Enforcement, & Values01:25:18 Closing & ThanksMentionsAndrea’s Website (which may not work; she’s extremely busy doing real stuff)Andrea’s Substack (which is fallow due to aforementioned real stuff but maybe a surge of new followers will inspire her to publish more of her excellent writing here)Kate Raworth - Doughnut EconomicsJack D. Forbes - Indigenous Spirituality & EthosYancey Strickler - Postcapitalism for RealistsThe Consilience Project - Technology is Not Values Neutral: Ending the Reign of Nihilistic DesignHenry’s Zoo - The Limits and “Good” of Public GoodsJames Bridle - Ways of Being This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.humansontheloop.com/subscribe