Building a Unified Data Platform at Pattern with William Graham
The orchestration of data workflows at scale requires both flexibility and security. At Pattern, decoupling scheduling from orchestration has reshaped how data teams manage large-scale pipelines.In this episode, we are joined by William Graham, Sen...
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI · The Data Flowcast
Audio is streamed directly from the publisher (audio-delivery.cohostpodcasting.com) as published in their RSS feed. Play Podcasts does not host this file. Rights-holders can request removal through the copyright & takedown page.
Show Notes
The orchestration of data workflows at scale requires both flexibility and security. At Pattern, decoupling scheduling from orchestration has reshaped how data teams manage large-scale pipelines.
In this episode, we are joined by William Graham, Senior Data Engineer at Pattern, who explains how his team leverages Apache Airflow alongside their open-source tool Heimdall to streamline scheduling, orchestration and access management.
Key Takeaways:
00:00 Introduction.
02:44 Structure of Pattern’s data teams across acquisition, engineering and platform.
04:27 How Airflow became the central scheduler for batch jobs.
08:57 Credential management challenges that led to decoupling scheduling and orchestration.
12:21 Heimdall simplifies multi-application access through a unified interface.
13:15 Standardized operators in Airflow using Heimdall integration.
17:13 Open-source contributions and early adoption of Heimdall within Pattern.
21:01 Community support for Airflow and satisfaction with scheduling flexibility.
Resources Mentioned:
https://www.linkedin.com/in/willgraham2/
Pattern | LinkedIn
https://www.linkedin.com/company/pattern-hq/
Pattern | Website
https://pattern.com
https://airflow.apache.org
https://github.com/patterninc/heimdall
https://netflix.github.io/genie/
Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.
#AI #Automation #Airflow #MachineLearning