Case Study: Goldsky scales blockchain data to 100 PiB and 100K+ partitions with WarpStream

The Road to 100PiBs and Hundreds of Thousands of Partitions Goldsky

The customer, Goldsky, a Web3 developer platform, faced significant challenges scaling their Apache Kafka infrastructure to handle their massive real-time and historical blockchain data streams. With over 7,000 topics and 49,000 partitions, their previous Kafka solutions were both prohibitively expensive, costing over half a million dollars annually, and unreliable, causing performance issues and customer churn. They required a robust, cost-effective streaming data layer.

WarpStream provided a drop-in Kafka replacement built on object storage, which decoupled hardware costs from partition counts and delivered tiered storage without performance penalties. The solution reduced Goldsky's total cost of ownership by over 10x, eliminated networking costs, and enabled their cluster to scale automatically. With WarpStream, Goldsky reliably grew their deployment to 3.7 PiB of data and 41,000+ partitions on their path to 100 PiBs.

Goldsky

Jeffrey Ling

Chief Technology Officer