Orchestrating Analytics and AI Workflows at Telia with Arjun Anandkumar
The future of data engineering lies in seamless orchestration and automation. In this episode, Arjun Anandkumar, Data Engineer at Telia, shares how his team uses Airflow to drive analytics and AI workflows. He highlights the challenges of scaling d...
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI · The Data Flowcast
Audio is streamed directly from the publisher (audio-delivery.cohostpodcasting.com) as published in their RSS feed. Play Podcasts does not host this file. Rights-holders can request removal through the copyright & takedown page.
Show Notes
The future of data engineering lies in seamless orchestration and automation. In this episode, Arjun Anandkumar, Data Engineer at Telia, shares how his team uses Airflow to drive analytics and AI workflows. He highlights the challenges of scaling data platforms and how adopting best practices can simplify complex processes for teams across the organization. Arjun also discusses the transformative role of tools like Cosmos and Terraform in enhancing efficiency and collaboration.
Key Takeaways:
(02:16) Telia operates across the Nordics and Baltics, focusing on telecom and energy services.
(03:45) Airflow runs dbt models seamlessly with Cosmos on AWS MWAA.
(05:47) Cosmos improves visibility and orchestration in Airflow.
(07:00) Medallion Architecture organizes data into bronze, silver and gold layers.
(08:34) Task group challenges highlight the need for adaptable workflows.
(15:04) Scaling managed services requires trial, error and tailored tweaks.
(19:46) Terraform scales infrastructure, while YAML templates manage DAGs efficiently.
(20:00) Templated DAGs and robust testing enhance platform management.
(24:15) Open-source resources drive innovation in Airflow practices.
Resources Mentioned:
https://www.linkedin.com/in/arjunanand1/?originalSubdomain=dk
Telia -
https://www.linkedin.com/company/teliacompany/
https://airflow.apache.org/
https://www.astronomer.io/cosmos/
https://www.terraform.io/
Medallion Architecture by Databricks -
https://www.databricks.com/glossary/medallion-architecture
Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.
#AI #Automation #Airflow #MachineLearning