Case Study: National Energy Research Scientific Computing Center (NERSC) achieves long-term, searchable hot-warm-cold time-series storage with Elastic

Using Elasticsearch to Manage a Supercomputer’s Hot, Warm, and Cold Architecture

The National Energy Research Scientific Computing Center (NERSC), the U.S. Department of Energy’s primary scientific computing facility serving more than 6,000 researchers, faced a major data challenge: how to manage and make accessible massive volumes of log and time-series metric data now and decades into the future for users running petaflop systems like Edison and Cori. Researchers need reliable, timely access to this data—sometimes immediately, sometimes years later—which required a single, scalable solution for hot, warm, and archival storage.

NERSC implemented Elasticsearch with a hot–warm–cold architecture and Curator 4: SSD-based hot nodes for speed, RAID5 + LVM-cached warm nodes for extended short-term storage, and GlusterFS backed by HPSS for permanent archival (some records exceed 30 years). With about 90 TB dedicated to Elasticsearch and a unified access method, NERSC achieved efficient retrieval, consistent long-term retention (they never delete HPSS archives), and simpler data management to support ongoing scientific analysis.

National Energy Research Scientific Computing Center (NERSC)

Thomas Davis

National Energy Research Scientific Computing Center (NERSC)