The Healthcare AI Podcast

AI in healthcare can save lives-or put them at risk. This episode explores guardrails, safety LLMs, regulation, and why generic AI controls fail in clinical settings.Timestamps:00:00 Introduction01:26 What are LLM guardrails and why do they matter in healthcare02:36 Why AI hallucinations are dangerous in medical settings03:47 Why people still use chatbots for medical advice05:13 Why generic AI safety tools fail in healthcare06:16 Regulation pressure: US vs Europe09:03 Guardrail frameworks: Guardrails AI, NeMo, Llama Guard15:08 Safety LLMs and red teaming medical AI22:17 Why healthcare AI needs application-specific testing27:49 Shift-left AI safety and responsible design32:44 The ELIZA effect37:27 Practical advice for teams building healthcare AI𝗥𝗲𝘀𝗼𝘂𝗿𝗰𝗲𝘀 𝗳𝗼𝗿 𝗟𝗶𝘀𝘁𝗲𝗻𝗲𝗿𝘀 ►Papers:- Hakim et al. (2025) "The need for guardrails with large language models in pharmacovigilance."- Meta's Llama Guard paper: "Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations" (arXiv:2312.06674)- Ayala-Lauroba et al. (2024) "Enhancing Guardrails for Safe and Secure Healthcare AI" (arXiv:2409.17190)Code and Models:- Hakim et al. analysis code: https://github.com/jlpainter/llm-guardrails/ - Llama Guard: Available on Hugging Face (requires approval)- gpt-oss-safeguard: https://huggingface.co/openai/gpt-oss-safeguard-20b (Apache 2.0)Medical Ontologies:- MedDRA (Medical Dictionary for Regulatory Activities): https://www.meddra.org/ - WHO Drug Dictionary: Via Uppsala Monitoring CentreRegulatory Guidance:- EMA AI Reflection Paper: https://www.ema.europa.eu/en/about-us/how-we-work/data-regulation-big-data-other-sources/artificial-intelligence - FDA AI Guidance: Available on FDA.govLISTEN ON ►YouTube: https://youtu.be/IWoARQ0G7sgApple Podcasts: https://podcasts.apple.com/us/podcast/the-healthcare-ai-podcast/id1827098175Spotify: https://open.spotify.com/show/2XNrQBeCY7OGql2jVhcP7tAmazon Music: https://music.amazon.com/podcasts/5b1f49a6-dba8-479e-acdf-9deac2f8f60e/the-healthcare-ai-podcastFOLLOW ►Website: https://www.johnsnowlabs.com/LinkedIn: https://www.linkedin.com/company/johnsnowlabs/Facebook: https://www.facebook.com/JohnSnowLabsInc/X (Twitter): https://x.com/JohnSnowLabs#HealthcareAI #AIGuardrails #MedicalAI #AISafety #AIEthics #HealthTech #AIRegulation #DigitalHealth #AIinMedicine #MLOps #AICompliance #AIHallucinations

Eugene Ugur Sayan is the Founder, CEO and President of Softheon, a technology company that powers healthcare.gov and state ACA exchanges serving over 10 million Americans daily. Eugene filed a patent on intelligent software agents in 1998, long before anyone was discussing "agentic" workflows. Over the years, they built the Massachusetts Connector (America's first state health exchange) and now power the infrastructure behind healthcare.gov, serving over 10 million Americans with 1,300+ AI agents running 24/7/365.This isn't theoretical AI. It's production systems where 99% accuracy means people lose health coverage. Eugene explains why healthcare demands "airline industry standards" (99.999% uptime), the PPT Framework (People, Process, Technology), how his team orchestrates agents across federal and 50-state AI regulations, and why Softheon owns its entire stack, from data centers to application layer technology.Timestamps:00:00 Opening & Introduction02:22 The 1998 Patent: Building agentic workflows before ChatGPT06:40 Why Healthcare AI Requires 99.999% Accuracy10:00 Autonomy, Alignment & Accountability Framework12:26 1,300 Semi-Autonomous Agents & Human-in-the-Loop (HITL)18:49 Three Layers of AI: Hardware, Platform, Applications23:09 Biggest Challenges: People, Process & Technology31:04 Breaking Innovation Barriers in Conservative Healthcare35:44 Transparency Rules & Value-Based Care39:08 The ICHRA Revolution: Healthcare's 401K Moment44:00 Consumer Engagement: Three Pillars Strategy50:00 Entrepreneurship Philosophy & Daily Practice53:00 Final AdviceListen On:YouTube: https://youtu.be/IWoARQ0G7sgApple Podcasts: https://podcasts.apple.com/us/podcast/the-healthcare-ai-podcast/id1827098175Spotify: https://open.spotify.com/show/2XNrQBeCY7OGql2jVhcP7tAmazon Music: https://music.amazon.com/podcasts/5b1f49a6-dba8-479e-acdf-9deac2f8f60e/the-healthcare-ai-podcastYou can learn more about the All-in-One Solution for Health Plans at Softheon.comFollow Eugene Sayan:LinkedIn: https://www.linkedin.com/in/eugenesayan/Twitter: https://x.com/SayanEugeneConnect With Us:Website: https://www.johnsnowlabs.com/LinkedIn: https://www.linkedin.com/company/johnsnowlabs/Facebook: https://www.facebook.com/JohnSnowLabsInc/X (Twitter): https://x.com/JohnSnowLabs#AI #HealthcareAI #AgenticAI #HealthTech #HealthcareInnovation

In this episode of The Healthcare AI Pod, we unpack the impact of LLM-based medical agents on modern medicine – from architecture and multi-agent design to regulation and real-world risks.Healthcare is facing a perfect storm: ageing populations, staff shortages, and rising costs. Can AI agents be the solution?We discuss insights from over 60 studies on medical LLMs, including key areas such as:Multi-agent architectures and clinical decision supportThe security dilemma: protecting patient data when your API is just textPrompt injection attacks and HIPAA compliance challengesLiability concerns in AI-powered healthcareFrom Baymax dreams to real-world implementation: how close are we?Timestamps0:00 Introduction – Baymax as inspiration for medical AI2:20 What are LLM-based medical agents and how they differ from models10:00 Healthcare security – regulation, compliance, and patient data14:50 Patient reliance on AI, prompt-hacking, and global access challenges18:00 Agent architectures – functional, role-based, and departmental approaches25:10 Task decomposition and subject-matter expert input28:00 Reward functions, accuracy vs user-pleasing bias, and physician training33:00 User experience – agent personalities and conversational design45:20 Liability, insurance, and evaluation of medical AI systems54:20 Future outlook – Baymax revisited, challenges, and opportunities aheadMentioned MaterialsA Survey of LLM-based Agents in Medicine: How far are we from Baymax? https://arxiv.org/abs/2502.11211MAGDA: Multi-agent guideline-driven diagnostic assistance https://arxiv.org/abs/2409.06351 Listen OnYouTube – https://youtu.be/R9h_Whj6sB0Apple Podcasts – https://podcasts.apple.com/us/podcast/the-healthcare-ai-podcast/id1827098175Spotify – https://open.spotify.com/show/2XNrQBeCY7OGql2jVhcP7tAmazon Music – https://music.amazon.com/podcasts/5b1f49a6-dba8-479e-acdf-9deac2f8f60e/the-healthcare-ai-podcastConnect With UsOur Website – https://www.johnsnowlabs.com/LinkedIn – https://www.linkedin.com/company/johnsnowlabs/Facebook – https://www.facebook.com/JohnSnowLabsInc/X (Twitter) – https://x.com/JohnSnowLabs#HealthcareInnovation #AIAgents #HealthTech #MedicalAI #AIEthics #Baymax #MedicalLLM #HealthcareAI #ClinicalAI #MedicalTechnology #AIResearch #DigitalHealth #FutureOfMedicine #AIinMedicine #HealthcareAutomation #MedicalChatbots #PatientCare #HealthcareSolutions #MedicalInnovation

Can AI make healthcare feedback fairer and smarter? In Episode 4 of The Healthcare AI Podcast, Ben Webster (VP of AI Solutions at NLP Logix) and David Talby (CEO of John Snow Labs) dive into a game-changing approach to AI governance. Discover how LangTest tackles bias in processing 1.5M hospital feedback audio files annually, ensuring fair sentiment analysis and actionable insights. From eliminating gender bias in nurse vs. doctor feedback to building robust, ethical AI models, this episode is a must-watch for healthcare and AI innovators! Join the Conversation: What’s the biggest challenge in healthcare AI today? Comment below!Timestamps06:18 – Bias in patient-feedback NLP 07:13 – LangTest & synthetic debiasing 12:30 – Data contamination & custom benchmarks 15:19 – Robustness testing & augmentation 20:18 – Medical red-teaming & safety checks 23:44 – Clinical cognitive biases in LLMsListen on your favourite platform: • ⁠YouTube⁠: https://www.youtube.com/playlist?list=PL5zieHHAlvApZKkwtu746ivthRc5zyTiU• ⁠⁠Apple Podcast⁠⁠: https://podcasts.apple.com/us/podcast/the-healthcare-ai-podcast/id1827098175• ⁠⁠Spotify⁠⁠: https://open.spotify.com/show/2XNrQBeCY7OGql2jVhcP7t • Amazon Music⁠⁠: https://music.amazon.com/podcasts/5b1f49a6-dba8-479e-acdf-9deac2f8f60e/the-healthcare-ai-podcast Connect with us: Our website: https://www.johnsnowlabs.com/ LinkedIn: https://www.linkedin.com/company/johnsnowlabs/ Facebook: https://www.facebook.com/JohnSnowLabsInc/ X: https://x.com/JohnSnowLabs #AIinHealthcare #AIBias #EthicalAI #AIGovernance #NLP #HealthTech #PatientFeedback #HealthcareAI

Dive into Episode 3 of the Healthcare AI Podcast, where Vishnu Vettrivel and Alex Thomas explore the growing world of Model Context Protocol (MCP) with a focus on Healthcare MCP (HMCP) from Innovaccer. This episode breaks down the essentials of MCP, from converting papers to N-Triples to deploying on Claude Desktop. Learn about resources, prompts, and tools that empower AI models, plus key security considerations. Stick around for a call to action to spark your thoughts on agentic frameworks! Tune in to discover why MCP could be the next big leap for AI in Healthcare. Timestamps 01:01 – Introducing the Model Context Protocol (MCP): Purpose & Core Concepts 05:44 – Healthcare MCP (HMCP) by Innovaccer 06:50 – Basics of MCP: Resources, Prompts, Tools 10:50 – Demo: Deploying to Claude Desktop (Example MCP Project) 22:24 – Healthcare Relevance & HMCP 23:46 – Security & Limitations 27:30 – Future Directions Listen on your favourite platform: • YouTube: https://www.youtube.com/playlist?list=PL5zieHHAlvApZKkwtu746ivthRc5zyTiU • ⁠⁠Apple Podcast⁠⁠: https://podcasts.apple.com/us/podcast/the-healthcare-ai-podcast/id1827098175• ⁠⁠Spotify⁠⁠: https://open.spotify.com/show/2XNrQBeCY7OGql2jVhcP7t • Amazon Music⁠⁠: https://music.amazon.com/podcasts/5b1f49a6-dba8-479e-acdf-9deac2f8f60e/the-healthcare-ai-podcastResources: - Model Context Protocol: https://modelcontextprotocol.io/overview - Introducing HMCP: A Universal, Open Standard for AI in Healthcare: https://innovaccer.com/resources/blogs/introducing-hmcp-a-universal-open-standard-for-ai-in-healthcare - We built the security layer MCP always needed: https://blog.trailofbits.com/2025/07/28/we-built-the-security-layer-mcp-always-needed/ Connect with us: Our website: https://www.johnsnowlabs.com/ LinkedIn: https://www.linkedin.com/company/johnsnowlabs/ Facebook: https://www.facebook.com/JohnSnowLabsInc/ X: https://x.com/JohnSnowLabs #MCP #ModelContextProtocol #HealthcareAI #MedicalData #AgenticAI #ClinicalAI #DataScience #HealthTech