Case Study: Zephyr AI achieves scalable, observable data science pipelines with Dagster Labs

Zephyr AI processes 80GB genetic files with Dagster Labs

Zephyr AI, a company specializing in AI-driven precision medicine, faced the challenge of orchestrating complex data pipelines for massive, disparate datasets including genomic sequences and electronic health records. Different teams used various tools, making collaboration and managing data lineage difficult. To bring structure and enable one-click deployments, they implemented a data orchestration platform from vendor Dagster Labs, using Dagster.

The solution using Dagster provided a centralized Python-based framework with software-defined assets, offering a single pane of glass for observability. This allowed Zephyr AI to seamlessly integrate diverse tools like Perl scripts and Scala programs, deploy on Kubernetes for scalable processing, and significantly accelerate pipeline development. The team achieved greater transparency, improved collaboration with product teams through clear data lineage, and gained performance benefits by efficiently scaling jobs, allowing them to focus on data analysis rather than infrastructure.

Zephyr AI