Case Study: JetBrains achieves scalable analysis and actionable dashboards from millions of NuGet packages with Elastic (Elasticsearch & Kibana)

How JetBrains uses .NET, Elasticsearch, CSVs, and Kibana for awesome dashboards

JetBrains set out to analyze the NuGet ecosystem to better understand open-source package trends, but the effort produced a massive dataset (a 1.5 GB CSV representing ~3.3 million records and over 231k unique packages). The challenge was reliably retrieving, cleaning, and processing millions of package records (authors, versions, tags, publish dates, download counts, etc.) while handling malformed rows, memory pressure, and tooling limits that made spreadsheets unusable.

Their solution used what they call the NECK stack—.NET, Elasticsearch, CSV, and Kibana—building a .NET console pipeline with CsvHelper to parse and normalize rows, NEST to define mappings and stream documents to Elasticsearch in bulk, and Kibana to create visualizations and dashboards. The team indexed millions of records in minutes (peaking at ~9 GB memory on a laptop), lost only a handful of malformed rows, and produced dashboards showing package counts, target frameworks, authorship, licenses, and time-based trends to drive their research and reporting.

JetBrains