Case Study: Zephyr AI accelerates precision medicine data pipelines with Dagster Labs

Orchestrating Data Science at Zephyr AI

Zephyr AI, a company applying data science to massive DNA and healthcare datasets for precision medicine, faced the challenge of orchestrating complex data science pipelines across different teams, tools, and enormous datasets. Their disparate bioinformatics and predictive analytics pipelines required a unified control plane for observability, data lineage, and collaboration, which their initial manual processes and naive schedulers could not provide.

By adopting Dagster early on, Zephyr AI implemented a centralized, Python-based orchestration framework. Dagster provided a single pane of glass for observability, enabled the use of software-defined assets for declarative pipeline management, and allowed dozens of teams to work in parallel without conflict. This solution accelerated development, provided crucial transparency for high-stakes clinical models, and allowed the team to seamlessly integrate diverse tools like Perl, Shell, and Java into their pipelines, ultimately letting them focus on data insights rather than fighting their tools.

Zephyr AI

Jeff Sherman

Chief Technology Officer