Case Study: Fireworks AI achieves 1 TB/s+ model deployment throughput with Alluxio

Fireworks AI cuts model load times to 1 minute with Alluxio

Fireworks AI, a leading inference cloud provider for generative AI, faced significant challenges with model cold starts across their multi-cloud GPU infrastructure. Synchronized deployments of large models (70GB to 1TB+) across hundreds of GPU replicas would saturate object storage bandwidth and trigger rate limits, causing model load times to spike from minutes to hours. This left expensive GPU clusters idle and created substantial operational overhead. To address this, they implemented a distributed data caching solution from Alluxio.

Alluxio provided a co-located caching tier on the GPU nodes themselves, leveraging local NVMe SSDs. This solution acted as a high-performance data acceleration layer, fetching each model from object storage only once per cluster before serving it at line rate to all concurrent GPUs. As a result, Fireworks AI achieved over 1 TB/s aggregate throughput, reduced model load times from hours to 1-3 minutes, and saw a 50% reduction in egress costs. The implementation eliminated cold start delays, freed up engineering resources, and significantly improved customer experience.

Fireworks AI