Episode 193
Ep193: The Conductor Behind Your Data Orchestra: Astronomer's Approach to AI Pipeline Management
AWS for Software Companies Podcast · Nate Goyer
February 10, 202617m 1s
Audio is streamed directly from the publisher (rss.art19.com) as published in their RSS feed. Play Podcasts does not host this file. Rights-holders can request removal through the copyright & takedown page.
Show Notes
Astronomer's Steven Hillion reveals how OpenAI, Anthropic, Uber, and Lyft use Apache Airflow to orchestrate AI and machine learning pipelines at scale on AWS.
Topics Include:
- Steven Hillion leads data and AI at Astronomer
- Apache Airflow surpassed Spark and Kafka in community metrics
- Astronomer coordinates data flow like conductor orchestrating instrumental platforms
- Organizations with data engineering teams use Airflow at scale
- Customers already used Airflow for ML before official promotion
- Uber and Lyft orchestrate pricing models using Airflow
- Astronomer runs on AWS with close integration partnerships
- OpenAI Anthropic and GitHub Copilot use Airflow for operations
- Internal data team uses Airflow creating feedback loops
- Evolved from constrained AI reports to agentic workflows
- Platform monitors generative AI output quality at user interactions
- Metadata and context increasingly critical for AI applications
- Learn more at Astronomer’s Data FlowCast podcast
Participants:
- Steven Hillion – SVP, Data and AI, Astronomer
See how Amazon Web Services gives you the freedom to migrate, innovate, and scale your software company at https://aws.amazon.com/isv/
Topics
Generative AI#AWSforSoftwareAgentic AIAIcloud computing providersawsAmazon.comcloud servicesAmazoncloud computingcloud serviceAnthropicastronomerUberOpenAilyft