Introducing Airflow 3.2
We introduce Airflow 3.2 and its updates for teams that build and operate data pipelines.Astronomer’s Head of Customer Education, Marc Lamberti, and Senior Manager of Developer Relations, Kenten Danas, break down what’s new, from asset partitioning...
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI · The Data Flowcast
Audio is streamed directly from the publisher (audio-delivery.cohostpodcasting.com) as published in their RSS feed. Play Podcasts does not host this file. Rights-holders can request removal through the copyright & takedown page.
Show Notes
We introduce Airflow 3.2 and its updates for teams that build and operate data pipelines.
Astronomer’s Head of Customer Education, Marc Lamberti, and Senior Manager of Developer Relations, Kenten Danas, break down what’s new, from asset partitioning to Async Python tasks and DAG versioning. They explore how these updates improve scheduling, performance and observability in production workflows.
Key Takeaways:
00:00 Introduction.
02:10 Airflow 3 architecture separates workers from the metadata database.
03:05 Plugin versioning and UI-based backfills simplify operations.
06:20 Asset partitioning enables granular, partition-level scheduling.
07:15 Triggering DAGs on partitions instead of full datasets.
11:05 Deferrable operators reduce worker slot usage.
12:00 Async operators reduce database pressure and overhead.
14:10 Async improves throughput, not single task speed.
22:20 Inlets and outlets improve asset lineage visibility.
23:00 DAG version markers show changes directly in the UI.
Resources Mentioned:
https://www.linkedin.com/in/marclamberti/
https://airflow.apache.org/
Astronomer | LinkedIn
https://www.linkedin.com/company/astronomer/
Astronomer | Website
https://www.astronomer.io/
https://www.astronomer.io/events/webinars/introducing-airflow-3-2-video
https://www.astronomer.io/docs/learn/airflow-partitioned-runs
https://www.astronomer.io/docs/learn/deferrable-operators
https://airflow.apache.org/docs/apache-airflow/stable/release_notes.html#airflow-3-2-0-2026-04-07
https://airflow.apache.org/registry/
Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.
#AI #Automation #Airflow #MachineLearning