Case Study: Adobe Experience Platform scales complex workflow orchestration with Apache Airflow

Adobe Experience Platform - Customer Case Study

The customer Adobe Experience Platform required a sophisticated orchestration service for its complex hierarchical data workflows, which needed to scale and be triggered on schedules or by events. Their challenge was building a system to author, schedule, and monitor these modern big data pipelines for both Spark and non-Spark jobs.

To solve this, Adobe Experience Platform architected its service using the execution engine from the vendor Apache Airflow. This solution leveraged Airflow's extensibility, its Kubernetes Executor for scaling, and its rich web UI for insights. The results allowed the platform to run thousands of concurrent workflows and offload all scheduling, dependency management, and retry logic to Apache Airflow, letting internal teams focus on business use cases.

Adobe Experience Platform

Raman Gupta

Adobe Experience Platform