Reflections on a Decade of Data Engineering at Seattle Data Guy
Lessons from the past decade of data engineering reveal how much the ecosystem has changed and what has stayed surprisingly consistent.In this episode, Benjamin Rogojan, Owner and Data Consultant at Seattle Data Guy, joins us to reflect on how the ...
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI · The Data Flowcast
Audio is streamed directly from the publisher (audio-delivery.cohostpodcasting.com) as published in their RSS feed. Play Podcasts does not host this file. Rights-holders can request removal through the copyright & takedown page.
Show Notes
Lessons from the past decade of data engineering reveal how much the ecosystem has changed and what has stayed surprisingly consistent.
In this episode, Benjamin Rogojan, Owner and Data Consultant at Seattle Data Guy, joins us to reflect on how the data engineering landscape has evolved alongside Apache Airflow. We explore when Airflow makes sense as an orchestrator, why batch processing is still dominant and how AI is reshaping the workflows and responsibilities of modern data engineers.
Key Takeaways:
00:00 Introduction.
03:00 Airflow becomes valuable when workflows involve many pipelines, teams and dependencies.
05:00 Data engineers are still focused on making data accessible and aligning work with business needs.
05:30 Batch pipelines remain the most common approach even as real-time use cases grow.
07:45 Many “real-time” requests are actually event-driven batch workflows.
09:00 Airflow replaced many custom-built pipeline systems with built-in dependency management.
11:00 Modern orchestration tools often build on Airflow concepts or differentiate from them.
14:00 AI can assist with writing SQL and pipelines but still requires experienced engineers.
15:30 Organizations are collecting increasingly granular data creating more engineering demand.
19:00 The data stack has shifted rapidly from Hadoop-era systems to modern cloud platforms.
Resources Mentioned:
https://www.linkedin.com/in/benjaminrogojan/
https://www.linkedin.com/company/seattle-data-guy/
https://airflow.apache.org
Airflow Summit / Airflow Conference
https://airflowsummit.org
https://www.snowflake.com
https://developers.hubspot.com
https://mlflow.org
Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.
#AI #Automation #Airflow