Case Study: Blizzard achieves near-real-time, billion-message scalability and rapid insights with Elastic

Blizzard Building a Near Real-Time Data Pipeline

Blizzard Entertainment reworked its telemetry platform to meet the explosive growth of game and client data. What began as a server-focused RMQ + Hadoop/MapReduce data lake struggled with schema drift, RMQ scaling, and slow batch analytics; the team set aggressive goals to collect data from everywhere and process tens to eventually hundreds of billions of messages per day while giving engineers and the business near-real-time access to usable metrics.

The solution migrated ingestion to Kafka, added enrichment pipelines, and layered Elasticsearch + Kibana (with Tribe nodes) for instant search and dashboards, alongside a required Schema Registry, a Telemetry Development Kit, and multi-language SDKs to enforce "What You Registered Is What You Get." The new pipeline enabled centralized, near-real-time debugging and business insight (e.g., CCU and network quality), dramatically increased scalability (hundreds of TB in ES, petabytes in HDFS), reduced reliance on MapReduce, and earned strong business adoption, while work continues on multi-cluster operations and logging/metrics improvements.

Blizzard

Chris Burkhart

Principal Technical Lead