Case Study: npm (The Package Manager for Node.js) achieves millisecond ad-hoc analysis of 7 billion log-file rows with OmniSci

Analyzing Billions of Rows of Log File Data

npm, the package manager for Node.js used by millions of developers, needed to analyze massive request logs — roughly 700 million events per day and about 10 days of retention (~7 billion rows) — to answer unpredictable ad‑hoc queries and perform operational diagnostics (package usage, Node/npm versions, PoP, IP anomalies). To meet this challenge they deployed OmniSci’s analytics database to explore request log data in production (running on an Amazon EC2 r3.8xlarge).

OmniSci’s GPU‑accelerated solution lets npm run millisecond‑latency ad‑hoc queries across multi‑billion‑row datasets on a single server with no indexing required. The implementation handles hourly bulk loads and enables real‑time trend analysis and fast diagnostics while reducing infrastructure and administration costs, giving npm lightning‑fast response times and minimal operational overhead.

npm

Laurie Voss

Co-founder and Chief Data Officer