Case Study: Empirico achieves large-scale multi-cloud data orchestration with Dagster Labs

Empirico uses Dagster to orchestrate trillions of rows across multi-cloud environments

Empirico, a drug discovery company, needed to enable its data scientists to run large-scale analyses without managing the underlying multi-cloud infrastructure, which was complex due to strict data regulations. They sought a modern orchestration tool to replace a cumbersome system and dynamically distribute compute workloads across different environments. They chose the Dagster data platform to address this challenge.

Using Dagster's StepLauncher abstraction, the Empirico team built a solution that conditionally routes computations to the appropriate cloud environment, such as Databricks on AWS, based on data sovereignty rules. This abstracted infrastructure concerns from scientists, centralized control for engineers, and ensured regulatory compliance. Dagster provided the dynamic capabilities needed for their workflows, leading to a successful platform that can introduce new compute providers without disrupting users.

Empirico