Astronomer's Steven Hillion reveals how OpenAI, Anthropic, Uber, and Lyft use Apache Airflow to orchestrate AI and machine learning pipelines at scale on AWS.Topics Include:Steven Hillion leads data and AI at AstronomerApache Airflow surpassed Spark and Kafka in community metricsAstronomer coordinates data flow like conductor orchestrating instrumental platformsOrganizations with data engineering teams use Airflow at scaleCustomers already used Airflow for ML before official promotionUber and Lyft orchestrate pricing models using AirflowAstronomer runs on AWS with close integration partnershipsOpenAI Anthropic and GitHub Copilot use Airflow for operationsInternal data team uses Airflow creating feedback loopsEvolved from constrained AI reports to agentic workflowsPlatform monitors generative AI output quality at user interactionsMetadata and context increasingly critical for AI applicationsLearn more at Astronomer’s Data FlowCast podcastParticipants:Steven Hillion – SVP, Data and AI, AstronomerSee how Amazon Web Services gives you the freedom to migrate, innovate, and scale your software company at https://aws.amazon.com/isv/

Ask this episode

“What did the council decide on item 42?” — answered with the speaker and the moment.

Transcript

Related