Case Study: Klaviyo scales real-time analytics with Ververica's Apache Flink

Stream processing with High Cardinality and Large State at Klaviyo

Klaviyo, a data-driven ecommerce marketing company, needed to scale a real-time analytics system that processes billions of events, deduplicates over a million events per second, and manages more than 1.5TB of state with very high cardinality and fan-out. Its earlier custom Python, RabbitMQ, Redis, and Cassandra-based pipeline had idempotency, accuracy, isolation, and performance issues, so the company evaluated stream-processing options and chose Ververica’s Apache Flink-based approach.

Ververica’s Apache Flink helped Klaviyo build a new streaming application called Abacus, using Flink state management and RocksDB to handle large state at scale. The new system improved reliability and scalability, supported near-real-time processing within one second, and reduced disk read IOPs from as high as 9,000 to the low hundreds, with only occasional spikes to around 1,000 during compactions. Klaviyo also moved away from Cassandra counters, improving overall performance and enabling a more accurate, resilient analytics pipeline.

Klaviyo

Ning Shi

Klaviyo