Case Study: Empirico achieves large-scale multi-cloud computing with Dagster Labs

Abstracting away infrastructure concerns in large-scale computing with conditional multi-cloud processing

Empirico, a drug discovery and development company, faced challenges running large-scale data analyses across multi-cloud and restricted computing environments due to data sovereignty regulations. Their team of data scientists required a solution that could abstract away complex infrastructure concerns and dynamically orchestrate workflows without them worrying about where the processing occurred. They turned to the vendor Dagster for a modern orchestration solution.

By implementing Dagster, Empirico built a data platform that uses conditional branching and a custom StepLauncher to dynamically route computations to the most appropriate cloud environment. This solution, provided by Dagster, centralized control for the engineering team, ensured compliance with data regulations, and abstracted the infrastructure layer from data scientists. This transformed their analytical capabilities, allowing them to seamlessly introduce new compute providers without disrupting existing workflows.

Empirico

Zachary Romer

Senior Data Infrastructure Engineer