Discover
Embedded AI Podcast
26 Episodes
Reverse
Ryan and Luca welcome back Sebastian Boblest and Marco Mader Mendes from Bosch to discuss the challenges of deploying AI in safety-critical embedded systems. The conversation explores two key aspects: ensuring the generated code itself is safe (following standards like ISO 26262), and guaranteeing the trained model behaves correctly in production (covered by ISO PAS 8800).The guests share practical strategies including fallback algorithms, model monitoring, redundancy approaches, and the importance of constraining what AI systems can do. They discuss real-world challenges teams face—from organizational resistance to unrealistic expectations about quick AI wins. Sebastian and Marco emphasize that while getting to 80% accuracy is often fast, reaching production-ready robustness requires significant effort. The episode offers a grounded look at what it actually takes to deploy ML safely in automotive, aerospace, and other domains where failures can threaten human life.Key Topics:[03:30] Two dimensions of functional safety: safe code generation (ISO 26262) vs. model behavior guarantees (ISO PAS 8800)[08:45] Code generation strategies: MISRA compliance, fixed memory layouts, and multi-variant verification[15:20] Practical safety approaches: fallback algorithms, output limiting, and model monitoring for out-of-distribution detection[22:10] Fuzzing ML systems and the importance of trying to break your model during development[25:40] Redundancy strategies: multiple models, sensor diversity, and majority voting for higher safety levels (ASIL C/D)[32:15] Industry trends: proliferation of NPUs on tiny devices and the challenge of opaque hardware accelerators[36:50] Common struggles: organizational resistance, unrealistic expectations, and the 80-to-99% problem in AI deploymentNotable Quotes:"You have to really think what you do. There's no off-the-shelf strategy. You have to really think about how you ensure that everything is working correctly." — Sebastian Boblest"We as AI engineers are not 'hey everything has to be replaced by AI.' Maybe it's better to have a hybrid model where you still trust physical equations that have been working for 100 years." — Marco Mader Mendes"You can solve problems with AI to the 80% level very quickly. People are super thrilled with these quick results. But the last 10 to 20% to get to product level—that gets quite tough." — Sebastian BoblestResources Mentioned:ONNX Functional Safety Subgroup - Initiative defining an ONNX subset for functional safety applications, largely driven by French aerospace companiesISO 26262 - Automotive functional safety standard covering software development in safety-critical applicationsISO PAS 8800 - Standard addressing how to ensure AI models behave correctly after training and deploymentIEC 61508 - Top-level standard for software development in functional safety-critical applications across industriesMISRA Standard - Coding guidelines defining what to do and not do in safety-critical embedded softwareEmbedded AI Coder (ETAS) - Bosch's code generation tool for deploying ML models on embedded systems with functional safety supportAgile Embedded Podcast Slack - Community discussion channel where you can reach Ryan and Luca, with a dedicated sub-channel for Embedded AI topics
We talk with Mark Deutel and Dr. Axel Plinge from Fraunhofer IIS about the art and science of running AI models on resource-constrained devices. Mark walks us through his PhD work on getting neural networks to run on microcontrollers with just 256KB of memory - a feat that requires clever optimization tricks, from neural architecture search to instruction-level tuning. Axel shares insights on why edge AI matters for robotics, where battery life and response time make local processing essential.The conversation ranges from their multilingual airport robot (complete with culturally appropriate gestures) to the practical challenges of teaching robots to grasp objects. We explore the shift from hand-engineered features to data-driven approaches, discuss why German privacy laws favor edge processing, and learn why the simplest microcontrollers sometimes give the most predictable performance. Mark and Axel make a compelling case that not every problem needs the biggest AI model - sometimes a specialized edge model is exactly the right tool.Key Topics:[03:30] Introduction to Fraunhofer IIS and its role bridging academia and industry[06:45] The multilingual airport robot: combining LLMs, speech processing, and local AI[15:20] Mark's PhD work: running neural networks on 256KB microcontrollers[19:40] The optimization pipeline: from neural architecture search to SIMD instructions[28:15] Why edge AI matters for robotics: battery life, latency, and the human analogy[35:50] Visual Language Action models and the challenge of world models in robotics[42:30] The complexity of grasping: what humans take for granted that robots must learn[48:20] German privacy laws (DSGVO) and why they favor edge processingNotable Quotes:"You're trying to run something that is way more compute-heavy on something that shouldn't actually be able to run it. It's kind of like the magic that guys in the 90s had when they wanted to run 3D games on old computers." — Mark Deutel"You cannot optimize without leaving stuff out. Mark says it's a magic trick, but it's not that much magic - yes, he got YOLO running on a system, but he also removed a lot of categories. Specialization is always part of the deal." — Dr. Axel Plinge"When you start to think about it, it's actually insane what considerations you make when you grasp something. You consider: is it fragile? Where's the best point to pick it? Should I pick it at the top or bottom? Are my hands sweaty? There's all kind of stuff you do without really thinking about it." — Mark DeutelResources Mentioned:Fraunhofer IIS Efficient AI Team - Research group building AI solutions with fast MLOps workflow for embedded devicesHAI Store - Fraunhofer's platform for downloading and evaluating compressed AI modelsMark Deutel's Research Page - Personal homepage with details on PhD research about efficient neural networks on microcontrollersPIB Robot - 3D-printable educational humanoid robot from Incento used in the airport demonstrationNVIDIA Jetson Thor - Edge GPU system used for running local LLM and speech processing in the airport robotAgile Embedded Podcast Slack - Community discussion channel where you can reach Ryan and Luca, with a dedicated sub-channel for Embedded AI topics
We explore the resurgence of self-hosted LLMs and why they're worth considering despite the convenience of frontier models. From data privacy concerns to cost management and consistent performance, there are solid reasons to run your own models. We break down the hardware requirements (spoiler: it's all about memory bandwidth), compare different runtime environments from beginner-friendly LM Studio to production-ready VLLM, and discuss practical model choices. Luca shares his experience running dual RTX 3090s in his basement, while Ryan questions whether his 2004 Athlon is still viable (it might be). We also touch on the reality that local models lag frontier models by about six months—which means they're roughly where Claude Opus 4A was in February, and that's perfectly usable for many tasks.Key Topics:[02:30] Why self-host? Data privacy, cost control, and avoiding unpredictable changes from hosted providers[08:45] Memory bandwidth is the real bottleneck—why GPUs still win and what that means for your hardware choices[15:20] Hardware options: Mac Minis vs. discrete GPUs, and why second-hand RTX 3090s offer the best price-performance[22:10] Available models: Quen, Gemini, DeepSeek, and Mistral—what works well locally and the six-month lag behind frontier models[28:40] Runtime environments compared: llama.cpp for flexibility, LM Studio for ease of use, Olama for Docker-like experience, and VLLM for production[35:15] Practical considerations: context windows, model size vs. available VRAM, and why speed matters more than you think[40:30] Announcement: UddleSpec SDD framework now available, and the Agile Embedded Podcast is now Pragmatic EmbeddedNotable Quotes:"You need memory bandwidth. All of the memory. Every single token filters through all of those neural network layers—potentially 8 billion parameters per token." — Luca"The marketing behind these tools is very much geared toward 'don't worry what's happening behind the curtain.' But there are more efficient ways of solving the same problem without incurring the same cost." — Ryan"I really feel like the most rational choice is a second-hand 3090 still. They have 24 gigabytes of VRAM, they're nice and fast, and they have about half the bandwidth of a 5090—which is still pretty good and very much usable." — LucaResources Mentioned:llama.cpp - Command-line LLM runtime environment with broad hardware support and REST APILM Studio - User-friendly GUI for running local LLMs with built-in model downloader and recommendationsOlama - Docker-like interface for managing and running local LLMs, good for experimentationVLLM - Production-grade LLM runtime with excellent parallel request handling and memory efficiencyHugging Face - Repository with hundreds of thousands of models, quantizations, and variants available for downloadUddleSpec - Luca's new SDD framework addressing real-world workflow needs—now available on GitHubPragmatic Embedded Podcast - Sister podcast (formerly Agile Embedded) covering embedded systems developmentAgile Embedded Podcast Slack - Community discussion channel where you can reach Ryan and Luca, with a dedicated sub-channel for Embedded AI topics
Ryan and Luca pull back the curtain on large language models, explaining what's really happening when you chat with an AI. They break down tokenization, context windows, and the stateless nature of LLMs—revealing why these tools aren't actually thinking or remembering, just generating the most likely next token based on massive matrices of weights. The conversation covers practical implications like why long sessions deteriorate, how caching affects costs, and why that apologetic "I won't do it again" from your LLM is meaningless. They also tackle the misleading anthropomorphization in AI marketing and share frustrations with features like Claude's opaque memory system. If you've ever wondered why your LLM seems to forget instructions or why it confidently states nonsense, this episode explains the mathematical reality behind the conversational illusion.Key Topics:[02:30] Tokenization: How LLMs break language into mathematical units[08:45] LLMs as stochastic algorithms: Predicting the next most likely token[12:20] Training models: Billions of parameters and matrix multiplication[15:10] Quantization: Trading precision for memory efficiency[18:00] Context windows and token limits: Why size matters and costs money[22:15] Stateless processing: LLMs don't remember, they reprocess everything[26:40] Caching and timeouts: The hidden costs of pausing your session[30:00] Hallucinations aren't bugs—they're the default mode of operation[35:20] Attention mechanisms and why large context windows cause degradation[40:15] Claude's memory system: Well-intentioned but problematic in practice[44:30] Mixture of experts: Sparse vs. dense models and routing tokensNotable Quotes:"The LLM doesn't even know what truth is, so it can't lie to you. It's not lying to the truth either. It's bullshitting—it doesn't care which way is true or false." — Ryan"All LLMs know how to do is hallucinate. They just generate chains of tokens. If you're lucky, those chains have some connection to the real world and are actually helpful." — Luca"I wish it wasn't programmed to just lie to me. If the LLM says 'I won't do it again,' yes it will, because it has no memory of this incident. It will behave the same way tomorrow." — LucaResources Mentioned:Luca's Training and Consulting - Luca's website with links to AI and embedded systems training coursesTulip Tree Tech Emulator - Ryan's commercial emulator for embedded systems development (Raspberry Pi, STM, ESP, Microchip)Google's Attention Paper - The foundational 2013 paper introducing the attention mechanism that enabled modern LLMsOn Bullshit by Harry Frankfurt - Philosophical work distinguishing bullshitting from lying—relevant to understanding LLM outputsAgile Embedded Podcast Slack - Community discussion channel where you can reach Ryan and Luca, with a dedicated sub-channel for Embedded AI topics
We talk with Sebastian Boblest and Marco Mader Mendes from Bosch about the surprisingly challenging art of deploying AI models on resource-constrained embedded devices. They share insights from their work on ETAS Embedded AI Coder, a tool that generates optimized C code from neural network models for microcontrollers—sometimes with as little as 100 bytes of RAM available.The conversation covers practical strategies for model compression (often by factors of 100x or more), the counterintuitive benefits of float models over quantized ones on tiny devices, and why feature engineering still matters. Sebastian and Marco explain how they navigate the trade-offs between RAM, compute time, and accuracy, and why each project presents unique constraints—from confidential hardware specs to compiler quirks. They also discuss real-world applications, including Bosch's AI-powered wall scanner that uses radar and neural networks to detect cables in walls.Key Topics:[03:30] Why Bosch started exploring embedded AI on tiny hardware in 2020[06:45] Real-world application: AI-powered wall scanner using radar to detect cables[09:20] How ETAS Embedded AI Coder generates C code from neural network models[12:00] Deploying models with as few as 300 parameters and 100 bytes of RAM[16:30] Model compression strategies: squeezing networks down by 100x or more[21:15] When float models outperform quantized ones on tiny devices[28:00] Trading off RAM vs. compute time through code generation techniques[33:45] Benchmarking challenges with confidential hardware and compilers[40:20] AutoML and architecture search for constrained embedded targets[46:00] Free tool access for universities and opportunities for studentsNotable Quotes:"We go down to applications where we use 100 bytes of RAM and you can still do something useful with this on a Cortex-M0. It's very surprising how small you can get with neural networks." — Sebastian Boblest"Sometimes we have to squeeze it down not by one or two X, sometimes it's up to 100 X and more. This is really a regular task for us." — Marco Mader Mendes"One thing that's super counterintuitive for many people on these smaller devices is to go from a quantized model to a float model. It can actually help you. It's exactly the opposite of what people do on these larger systems." — Sebastian BoblestResources Mentioned:ETAS Embedded AI Coder - Code generation tool for deploying neural networks on embedded devices; free for universitiesARM CMSIS-NN - Library containing functions for neural network layers on Cortex-M devicesMLPerf Tiny Benchmarks - Benchmarking suite for tiny ML systems that Sebastian's team participated inBosch AI-powered Wall Scanner - Radar-based power tool using neural networks to detect cables in wallsAgile Embedded Podcast Slack - Community discussion channel where you can reach Ryan and Luca, with a dedicated sub-channel for Embedded AI topics




