Case Study: Guardant Health achieves scalable, high-performance genomics processing with Alluxio

Scalable Genomics Data Processing Pipeline with Alluxio, Mesos, and Minio

Guardant Health, the world leader in comprehensive liquid biopsy, needed a data processing platform that could handle exabyte-scale genomic workloads while remaining flexible, scalable, and highly performant. Its existing disk-based approach could not keep up with the speed and volume of genomic data, so the company looked for a solution that could work across on-premises and cloud storage systems. Alluxio was part of the stack used to address this challenge, alongside Mesos, Minio, and Spark.

Guardant Health implemented Alluxio as a unified storage layer to connect disparate storage systems, with Minio serving as the under storage and Apache Spark as the compute engine, all orchestrated with Mesos. This architecture enabled independent scaling of compute and storage, improved performance by leveraging memory aggressively, and created a cloud-native, cost-effective pipeline. According to the case study, the production deployment reached 50+ nodes, over 20 TB of Alluxio RAM capacity, more than 200 TB of HDD storage, and 1.2 PB+ of Minio archival storage, delivering orders-of-magnitude higher performance than before.

Guardant Health