Case Study: LINE achieves near-real-time, terabyte-scale log processing with Fluentd

From Batch to Stream Log Processing with Fluentd

LINE, the Tokyo‑based messaging and services platform, faced the challenge of collecting, storing and analyzing massive daily log volumes previously handled by a Scribe-to-Hadoop batch pipeline. To get near real‑time insight and free up Hadoop for other batch jobs, LINE sought an in‑stream data collector and adopted Fluentd to move fixed‑window processing into the data-collection layer.

Fluentd was extended and optimized in collaboration with LINE engineers, yielding a 10x–15x performance improvement and enabling LINE to process about 1.5 TB (5.6 billion records) of logs per day at peaks above 120,000 records/sec. LINE also developed 34 Fluentd plugins and a schemaless SQL stream engine (Norikra) on top of Fluentd, which together delivered real‑time analytics, reduced load on Hadoop, and faster internal decision‑making.

Line

Satoshi Tagomori

Line