Case Study: CERN processes petabytes more efficiently with Kubernetes

Processing Petabytes of Data More Efficiently with Kubernetes

CERN, the European Organization for Nuclear Research in Geneva, Switzerland, needed a way to process massive volumes of experimental data more efficiently as its storage already reached 330 petabytes and was expected to grow 10x with future accelerator upgrades. The organization also faced extreme workload spikes around major conferences and wanted a more hybrid infrastructure that could scale beyond its on-premises environment. To meet these challenges, CERN turned to Kubernetes for container orchestration.

With Kubernetes, along with Helm, Prometheus, CoreDNS, and Kubernetes federation, CERN built a more automated and flexible platform for batch workloads, Spark-based analytics, machine learning, and hybrid cloud execution. The impact was significant: deploying a complex distributed storage cluster dropped from more than 3 hours to under 15 minutes, adding nodes fell from over an hour to under 2 minutes, and autoscaling replicas also improved from more than an hour to under 2 minutes. Kubernetes also helped reduce virtualization overhead from 20% to about 5%, with a path to 0% on bare metal, while reclaiming an estimated 10% of memory capacity.

CERN

Ricardo Rocha

Software Engineer