Case Study: Anaconda scales package builds with Prefect

Anaconda scales package builds across 7 platforms with Prefect

Anaconda, a provider powering the modern data ecosystem, faced the challenge of building thousands of open-source packages across seven different platforms. This process was complex due to the diversity of programming languages, build systems, and the need to easily bootstrap new platforms. Their existing CI/CD tooling required significant work to adapt, making it difficult to manage this scale efficiently and provide timely updates and security fixes for their users.

To solve this, Anaconda implemented an event-driven build pipeline using Prefect Cloud. The solution leverages AWS infrastructure and Prefect's Python-based automation to orchestrate builds, manage a heterogeneous set of workers, and provide observability. The result is a highly scalable and reliable pipeline that offers great insight into the build process, ensuring Anaconda's large user base receives the up-to-date packages they need.

Anaconda