Case Study: Together AI achieves sub-second LLM observability with StarTree Cloud

Together AI launches real-time dashboards in 30 days with StarTree

Together AI, a frontier AI cloud platform, confronted a significant observability challenge as the token volume on its platform surged to billions per hour. Their existing analytics stack was unable to provide the real-time, granular insights needed by customers for tracking usage, engineers for debugging, and finance teams for accurate billing. To address the need for low-latency, high-cardinality analysis of LLM usage, they turned to the vendor StarTree and implemented its StarTree Cloud service.

StarTree Cloud, powered by Apache Pinot, was deployed and delivering results within 30 days. The solution provided sub-second query latency across billions of events, enabled high-cardinality slicing and dicing of data, and delivered data freshness in 10-second windows. An engineer from Together AI highlighted a dramatic performance improvement, noting query latency dropped from 10 seconds to just 7 milliseconds. This allowed the company to offer near-instant visibility into LLM usage, transforming its observability from a backend function into a core part of its product experience.

Together AI