Case Study: CloudFlare achieves real-time, sub-second analytics on 100s of TB with Citus Data's CitusDB

CloudFlare Powers Real-Time Analytics on 100s of TB of Data from Over 4 Million Websites with CitusDB

CloudFlare, which accelerates and protects millions of websites and handles a significant portion of global web traffic, outgrew its original log-processing pipeline (Perl/C++ and a single PostgreSQL instance) as data traffic surged over 400% annually. Its small analytics team needed an easy-to-operate, Postgres-compatible solution capable of returning customer queries in under a second across many sites and long time windows, without the heavy development effort required to manually shard Postgres.

They deployed CitusDB, a Postgres extension that distributes data and parallelizes queries across commodity servers, enabling real-time ingest and sub-second queries on billions of rows and roughly 100 TB+ of data (about 1 million shards). The cluster was live in under three months with the equivalent of 2.5 people, delivering 25 ms–2 s query latencies, fivefold reductions in development effort, an 85% drop in production costs, cross-customer BI for millions of sites, and high availability via shard replication and automatic failover.

Cloudflare

Albert Strasheim

Systems Engineer