Discover
System Prompt
System Prompt
Author: Peter
Subscribed: 0Played: 1Subscribe
Share
© 2026 Peter
Description
System Prompt is a podcast about what’s actually happening in AI.
Not hype. Not surface-level takes.
We break down how AI is changing software, SaaS, infrastructure, and the way systems are built focusing on real-world tradeoffs, architecture decisions, and where the value is actually shifting.
If you’re building, deploying, or thinking seriously about AI, this is for you.
34 Episodes
Reverse
Episode 34: The State of AI | Where’s the ROI?The models keep getting better. So why is it still so hard for businesses to see a return?In this off-the-cuff episode of System Prompt, Peter and Val catch up on frontier model releases, AI security, and the economics of running all this compute. The conversation gets practical when they ask what happens after a company buys AI licenses and someone has to figure out how the work actually gets done.Val talks about what it takes to get employees to explain workflows they’ve spent years figuring out on their own. Peter shares how he used a local model on his ROG Flow Z13 to build an invoice tool, with code handling repeatable steps and the model interpreting the invoices.They also get into hardware costs, consolidation among AI providers, and what open-weight models mean for people who want more control over where and how they run AI. Peter closes with a preview of his ticket-to-PR software factory and the decisions that went into building it.IN THIS EPISODE• Frontier model releases and AI-assisted security• Enterprise AI ROI, licensing, and rollout friction• Finding out how work actually gets done• GPU costs and the economics of specialized cloud providers• Running local models on a tablet• General-purpose assistants and task-specific agents• Open-weight models, competition, and access to compute• A preview of Peter’s ticket-to-PR software factoryTOOLS & RESOURCES• ASUS ROG Flow Z13 — Peter’s local AI setup• Qwen 27B and Gemma 4 — local models discussed in the episode• Hermes and Kokoro — local agent and text-to-speech toolsKEY TAKEAWAYSSTART WITH THE WORKFLOWAsk the people doing the work where the process slows down and what a good result looks like. Those answers give you something to build and measure.USE THE MODEL WHERE IT CAN HELPCode can handle repeatable steps and enforce checklists. A model can help interpret messy inputs, like invoices that come in different formats.LOCAL MODELS GIVE BUILDERS ANOTHER OPTIONRunning a model on hardware you own gives you more control over how you use it. Peter’s invoice tool is one example of what that looks like in practice.CHAPTERS00:00 Introducing the State of AI00:37 Frontier Model Releases and AI Security06:47 Why Enterprise AI ROI Is Hard to Measure11:13 How Work Actually Gets Done14:11 Checklists and Repeatable Workflows19:57 AI Valuations and the Chip Market33:20 GPU Cloud Economics and Local AI on a Tablet35:56 Building Agents Around Specific Tasks39:38 Where AI Stands Heading Into 202745:33 Coding as a Business Use Case52:15 Local and Frontier Models in a Software Factory55:47 Why Open-Weight Models Matter59:44 What Smaller Models Can Build1:01:33 Previewing the Ticket-to-PR Factory
Episode 33: AI Software Factories | The Future of Software Engineering?What happens when AI coding agents move beyond individual tasks and become part of a repeatable engineering process?In Episode 33 of System Prompt, Peter and Val explore AI software factories: systems designed to take engineering work from tickets through implementation, verification, review, and delivery.Drawing on AI DevCon NYC, Peter shares lessons about verification-aware programming, engineering standards, and deterministic guardrails. They discuss how preconditions, postconditions, and invariants can make specific software properties mathematically verifiable, reducing reliance on AI-generated tests and model judgment alone.But building a reliable factory introduces new challenges. How do you prevent agents from drifting away from specifications? Who makes architectural decisions? And can multiple agents coordinate without introducing more complexity than they solve?Val raises the economic question: do software factories actually reduce development costs, or do inference, orchestration, and supervision erase the savings?They close by examining greenfield development, the future of engineering teams, and how AI factories could reshape software development and SaaS businesses.IN THIS EPISODE• Coding agents vs. AI software factories• Lessons from AI DevCon NYC• Verification-aware programming and mathematical proofs• Engineering constitutions and specifications• Preventing agent drift and unreliable testing• Multi-agent orchestration and independent review• Ticket-to-PR workflows and human oversight• Open-source models, inference costs, and ROI• Greenfield development and future engineering teamsTOOLS & RESOURCESAI DevCon NYC:https://tessl.io/devconLemmaScript — Verification-aware programming:https://docs.lemmascript.org/Tessl — Agentic software engineering:https://tessl.io/KEY TAKEAWAYSTHE FACTORY IS MORE THAN THE AGENTReliable factories need repeatable workflows, engineering standards, authorization boundaries, and trustworthy verification.SPECIFICATIONS DRIVE THE OUTCOMEAgents can implement misunderstood requirements. Clear specifications and human judgment remain essential.VERIFICATION GOES BEYOND AI REVIEWFormal contracts can mathematically verify specified properties. A second AI reviewer is useful but cannot replace mechanical verification.THE HARNESS MATTERSTools, execution environments, context, and deterministic controls shape reliability as much as model capabilities.ECONOMICS DETERMINE ADOPTIONCheaper inference doesn't guarantee cheaper development. Cost per accepted task is what matters.ENGINEERING ROLES WILL EVOLVEArchitecture, design, verification, and domain expertise may become increasingly important as implementation becomes automated.CHAPTERS00:00 Introducing AI Software Factories01:52 What Is an AI Software Factory?04:43 Engineering Standards and Workflows07:28 Lessons from AI DevCon NYC09:57 Mathematical Verification and Contracts13:47 Verification-Aware Programming15:43 More AI DevCon Insights20:35 Engineering Challenges and Specifications23:42 Quality, Security, and Coordination27:00 Engineering Constitutions28:16 Optimizing Prompts and Skills30:56 Factory Workflows in Practice32:46 Starting with Ticket-to-PR35:03 The Software Factory Assembly Line38:22 Models and Coding Harnesses41:37 Do Factories Make Economic Sense?44:59 Cost per PR and Business ROI47:36 Model Selection and Cost per Task48:00 Greenfield Development50:53 The Future of Engineering and SaaS55:59 AI Infrastructure and Competition57:14 Final Thoughts
Episode 32: How to Pick the Right AI Model | DeepSeek vs Qwen EvalsYou've seen how an AI agent is built, configured, and given a job. But how do you know if the model behind it is right for the work?In Episode 32 of System Prompt, Peter and Val close out their agent series with a live demonstration of AI evaluations, or "evals." Instead of trusting public benchmarks or model reputation, they test the tasks that matter to a real workflow.Using smevals, they compare DeepSeek V4.1 Flash running locally across two NVIDIA DGX Sparks against Qwen 3.8 27B. Their tests cover structured data extraction, instruction following, reasoning, and coding.The smaller Qwen performs surprisingly well. But the real lesson comes from unexpected failures: DeepSeek adds a timezone to an otherwise correct answer, while Qwen initially returns no output on a task it passes when rerun. Peter and Val examine the outputs, question the grading rules, and troubleshoot live rather than presenting a polished happy path.They also explore how harnesses handle model behavior, why larger models aren't always necessary, and how to connect performance, hardware costs, and business ROI. The goal isn't to choose the biggest model. It's to choose one that can reliably do your job at a sensible cost.IN THIS EPISODE• Why benchmarks don't always represent your workload• How smevals uses tasks, runners, graders, and reports• DeepSeek V4.1 Flash vs Qwen 3.8 27B• Structured data, instruction following, reasoning, and coding tests• Diagnosing grading mismatches and missing responses• Why repeatability and failure severity matter• Harness engineering and deterministic validation• Local hardware costs, model selection, and business ROITOOLS & RESOURCESsmevals — Open-source evaluation framework:https://github.com/prime-radiant-inc/smevalsKEY TAKEAWAYSBENCHMARKS AREN'T YOUR WORKLOADTest against your real inputs, constraints, and expected outputs—not a leaderboard alone.NOT ALL FAILURES ARE EQUALA formatting mismatch, empty response, or strict grader doesn't automatically prove a model can't do the task. Investigate the cause.SMALLER CAN BE GOOD ENOUGHFor routine tasks, a 27B model may meet requirements that don't justify the cost of running a much larger model.EVALUATE THE EVALUATIONSGraders, runtimes, and model outputs can all vary. Rerun tests and examine failures before drawing conclusions.THE HARNESS MATTERSPrompts, tools, validation, and deterministic logic shape the reliability of the full system—not just the model.ROI STARTS WITH THE WORKFLOWUnderstand the task, its cost, and the value of automating it before deciding which model to deploy.CHAPTERS00:00 Closing the Agent Series: Picking a Model01:19 Why Evals Matter04:52 Introducing smevals07:33 DeepSeek vs Qwen09:25 Running the Evaluation Suite12:04 Structured Data Results14:23 Instruction-Following Failures15:49 Investigating a Grading Mismatch18:29 Failure Severity and Impact19:48 Deterministic Output Handling21:14 Harness Engineering23:03 Models vs Agent Harnesses25:19 Investigating Qwen's Missing Response26:28 Repeated Evaluations27:21 Rerunning Tests28:57 Human Bias vs Data29:41 Coding Results32:01 When Smaller Is Good Enough34:56 Productivity and ROI37:55 Understanding Business Problems40:09 Wrap-Up
You've seen what goes into an AI agent. Now what does it take to actually configure one and give it a job?In Episode 31 of System Prompt, Peter and Val move from the architecture of Hermes Agent to a hands-on walkthrough of its Windows desktop interface. With Hermes already installed and connected to DeepSeek V4.1 Flash running locally across two NVIDIA DGX Sparks, they explore the settings that shape an agent's behavior, access, memory, and execution.They work through model selection and fallbacks, chat personalities, workspace configuration, approvals, file checkpoints, context compression, providers, tool integrations, and plugins. Not every feature needs changing, and not every experiment works as expected. That's part of the demonstration: no prearranged happy path.Then they give Hermes a real task: create a scheduled 8 a.m. AI news briefing, add sentiment analysis, inspect sources and output, and request a more polished PDF format. The conversation expands into what makes an automation useful—from daily operational reporting and ad analytics to finding RFPs and preparing proposals. The question isn't just what an agent can do. It's which steps in your existing workflow are worth handing off.• Configuring Hermes through the Windows GUI• Primary models, reasoning settings, fallbacks, and auxiliary models• Personalities, attachments, interface settings, and workspaces• Command approvals, secret redaction, file checkpoints, and browser access• Persistent memory and context compression• Agent step limits, subagents, and local versus cloud providers• Search tools, custom integrations, MCP, skills, and plugins• Testing web access with a locally hosted model• Creating an automated AI news brief with sentiment analysis• Applying agents to real business workflowsKEY TAKEAWAYSCONFIGURATION IS PART OF AGENT ENGINEERINGChoosing a model is only the beginning. Workspace access, approval rules, memory, tools, and execution limits determine how an agent operates.LOCAL DOESN'T MEAN UNRESTRICTEDA locally hosted model can use web tools and run workflows, but access to browsers, files, credentials, and commands still needs deliberate controls.YOU DON'T NEED TO CHANGE EVERY SETTINGSome defaults are fine. Understand what each setting governs, then adjust it based on the task and the risk.A WORKING DEMO ISN'T A FINISHED EVALUATIONHermes creates a schedule and generates an initial briefing, but repeatability, source quality, and output consistency still need testing.START WITH YOUR WORKFLOW, NOT THE TOOLLook at what you retrieve, compare, update, email, and report each day. The valuable agent is the one that removes real work from that process.CHAPTERS00:00 From AI Infrastructure to a Working Agent02:29 Models, Reasoning, and Fallbacks03:30 Chat Personalities and Attachments05:39 Appearance and Desktop Preferences07:14 Workspaces, Approvals, and File Safety08:44 Memory, Compression, and Voice10:51 Advanced Settings and Agent Limits12:00 Providers, Local Models, and Gateways14:00 Tool Integrations, Plugins, and Skills16:55 Can Hermes Build Its Own Tools?18:24 Testing Web Access19:20 What Should You Actually Do With an Agent?21:12 Building a Scheduled AI News Brief28:18 Adding Sentiment Analysis30:58 Agent Personalities and Human Interaction33:45 Reviewing the Briefing and Sources37:20 Requesting a PDF Output39:01 RFPs and Business Automation Opportunities41:59 Finding Workflows Worth Automating47:05 Wrap-Up and What's Next
READ THE FULL EPISODE PAGE: https://devmesh.tech/podcast/anatomy-of-an-ai-agentAn AI agent isn't just a model with tools. What actually happens between your prompt and a completed task, and what keeps the system from falling apart when something goes wrong?In Episode 30 of System Prompt, Peter and Val take apart the Hermes Agent harness to examine the software engineering behind an AI agent. Building on their episodes about AI hardware and inference runtimes, they follow the agent loop from context assembly and model calls through tool execution, persistence, security, recovery, and observability. Hermes is the example, not the endorsement: the point is to show where probabilistic model decisions end and deterministic software takes over.They explore why switching model providers involves more than changing an API endpoint, why a tool call is a request rather than an action, and how seemingly ordinary components like SQLite, file-based memory, and approval rules support a sophisticated system. Along the way, they discuss evaluation, concurrency, prompt caching, local execution risks, and the trade-offs that come with every architecture.The agent loop may be simple. Making it durable, safe, and useful against real systems is the engineering challenge.• The model versus the harness: who decides what• The core agent loop and turn management• Context construction, caching, and token costs• Provider normalization across APIs and local models• Tool registries, validation, approvals, and execution• Persist-before-execute and crash recovery• Iteration limits, errors, and fallback behavior• Memory, skills, and what deserves to be remembered• Security boundaries and dangerous-command detection• SQLite, concurrency, observability, and evaluations• Hermes' strengths, limitations, and engineering trade-offsKEY TAKEAWAYSTHE MODEL REASONS. THE HARNESS GOVERNS.The model proposes tools, arguments, and responses. The harness determines what can run, where it runs, what gets persisted, and which actions require approval.A TOOL CALL IS NOT EXECUTIONThe model can request an action, but deterministic code validates the request and enforces permissions before a tool actually does anything.PROVIDER COMPATIBILITY IS MORE THAN AN ENDPOINTReasoning formats, message structures, streaming, and error handling differ. Normalizing those differences is substantial engineering work.RECOVERY IS BUILT FROM ORDINARY SOFTWARESession leases, database writes, iteration controls, and error handling make a simple model/tool loop survivable when processes crash or requests fail.SECURITY IS AN ENGINEERED BOUNDARYInstructions alone aren't permission controls. Approval states, restricted execution environments, and configuration determine what an agent can actually do.EVERY ARCHITECTURE HAS TRADE-OFFSHermes uses practical building blocks, but memory growth, provider complexity, local permissions, and SQLite concurrency still require deliberate choices.CHAPTERS00:00 Intro: Anatomy of an AI Agent04:15 Why Is the Hermes Repository So Large?06:33 Agent Loop, Turn State, and Context Assembly10:47 Specialized Harnesses and Context Costs12:35 Provider Abstraction and Tool Registries17:46 Who Actually Executes a Tool Call?19:41 Approvals, Failures, and Evaluations23:48 Recovery, Memory, and Skills27:35 Deterministic Security and Execution Boundaries31:21 Concurrency, SQLite, and Observability36:25 Testing, Prompt Injection, and Memory Decisions40:43 What the Model Decides vs. What the Harness Decides45:24 Hermes Limitations and Engineering Trade-Offs50:32 Final Takeaways: What Makes an Agent a System52:02 Wrap-Up and What's Next




