The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

124 Episodes

Reverse

Building AI for data engineering at Astronomer

2026-11-1226:27

Generic AI coding tools consistently fall short on Airflow work. Only 9% of users in last year's Airflow survey said they were satisfied with them. In this episode, Yetunde Dada, Senior Director of Product Management at Astronomer, and Stephanie Niu, Senior Product Manager at Astronomer, join Kenten to walk through Otto, Astronomer's data engineering agent built specifically for Airflow, and how it approaches investigations, upgrades, and evaluation. Otto is available now in Labs and will ent...

Orchestrating Qualcomm’s Semiconductor Chip Design At Scale

2026-11-0528:38

Semiconductor chip design is not where most people expect to find Apache Airflow, but Qualcomm's CPU team runs millions of pre-silicon simulations and post-silicon validation jobs through it every day. In this episode, Kenten Danas is joined by Dheeraj Turaga, Principal Engineer at Qualcomm and an Apache Airflow committer, to talk through two very different Airflow use cases: orchestrating semiconductor design workflows across on-prem compute farms, and driving robotic handlers in post-silico...

Orchestrating cloud data migrations at HCL Tech

2026-10-2920:03

Large-scale cloud migrations can result in a lot to manage when it comes to orhcestration: multi-cloud targets, on-prem extraction, dynamic workloads, and strict operational controls. In this episode, Kenten Danas is joined by Bhanu Prakash, Technical Lead at HCL Tech, to talk through how his team uses Airflow to orchestrate end-to-end data and platform migrations to the cloud, generate Dags dynamically for micro-batch workloads, and automate incident response through a ServiceNow integration...

Building self-service Dags with GitHub Actions at Lyft

2026-10-2229:58

At Lyft, a small data engineering team supports dozens of scientists and analysts who tune driver pay configurations on demand, all under SOX controls. In this episode, Ken Obata, Senior Software Engineer at Lyft, joins Kenten Danas to walk through how his team built self-service Dags with an event-driven integration between GitHub Actions and Airflow, and how they solved the data modeling and consistency problems that came with it. Key Takeaways: 00:00 Introduction. 02:49 How Airflow r...

Orchestrating cost-efficient data pipelines at Insignis

2026-10-1530:58

Running Airflow with a one-person data engineering team means every decision about cost, tooling, and best practices has to earn its keep. In this episode, Kenten sits down with Aniruddha Sengupta, Staff Data Engineer at Insignis, to talk through how the UK cash-savings fintech runs a low-cost, scalable Airflow platform. The conversation covers observability, DAG-level best practices, the internal tools Insignis built to control costs, and where AI fits into the roadmap. Key Takeaways: 00...

#box-pro-ellipsis-179468698932024{-webkit-line-clamp:2;}The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI