Case Study: Apple achieves large-scale ML data orchestration with Kestra

Apple's 200-engineer ML team orchestrates large-scale pipelines with Kestra

Apple, a global technology company, faced significant operational challenges with Apache Airflow while managing massive-scale data pipelines for its ML team. The complexity and overhead of maintaining Airflow diverted the focus of its 200 engineers from solving ML problems to managing infrastructure. They sought a new orchestration solution that could handle their scale without the high operational cost.

The company implemented Kestra, an orchestration platform, to manage its complex ETL workflows. Kestra's declarative YAML syntax and robust fault tolerance allowed the engineers to define pipelines easily and handle failures predictably. This solution enabled the team to reliably orchestrate data ingestion from sources like the App Store and Apple Music to their ML platform, freeing them to focus on coding for ML problems rather than infrastructure management.

Apple