Case Study: DigitalOcean achieves scalable centralized logging and real-time insights with Elastic (Elasticsearch & Kibana)

DigitalOcean - Customer Case Study

DigitalOcean’s Metrics & Logging team faced a classic cloud-scale logging problem: more than 10,000 systems across multiple datacenters and dozens of critical services with no centralized log aggregation. Engineers were forced to SSH and tail logs by hand, which didn’t scale—once aggregation began they discovered roughly 100,000 log lines/sec and that ~70% of traffic was noise from a single hypervisor program.

The team implemented regional rsyslog aggregators, indexed logs into Elasticsearch, and standardized log formats using CEE and liblognorm. They added Kibana dashboards, a CLI (Ummon), secure real-time tails (Logtalez), Atlantis integration, and ZeroMQ for flexible transport, then planned a purpose-built Elasticsearch cluster on droplets. The result: searchable, visualized logs with real-time tracing, billions of indexed documents (multi‑billion scale), dramatically reduced noise, faster troubleshooting, and a scalable foundation for alerting and analytics.

DigitalOcean

Brian Knox

Tech Lead