Building a Unified Data Platform at Pattern with William Graham
Description
The orchestration of data workflows at scale requires both flexibility and security. At Pattern, decoupling scheduling from orchestration has reshaped how data teams manage large-scale pipelines.
In this episode, we are joined by William Graham, Senior Data Engineer at Pattern, who explains how his team leverages Apache Airflow alongside their open-source tool Heimdall to streamline scheduling, orchestration and access management.
Key Takeaways:
00:00 Introduction.
02:44 Structure of Pattern’s data teams across acquisition, engineering and platform.
04:27 How Airflow became the central scheduler for batch jobs.
08:57 Credential management challenges that led to decoupling scheduling and orchestration.
12:21 Heimdall simplifies multi-application access through a unified interface.
13:15 Standardized operators in Airflow using Heimdall integration.
17:13 Open-source contributions and early adoption of Heimdall within Pattern.
21:01 Community support for Airflow and satisfaction with scheduling flexibility.
Resources Mentioned:
https://www.linkedin.com/in/willgraham2/
Pattern | LinkedIn
https://www.linkedin.com/company/pattern-hq/
Pattern | Website
https://pattern.com
https://airflow.apache.org
https://github.com/patterninc/heimdall
https://netflix.github.io/genie/
Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.
#AI #Automation #Airflow #MachineLearning
United States
