Case Study: Dippy AI achieves 4M+ tokens/min throughput with Together AI Dedicated Endpoints

Dippy AI reaches 4.1M tokens/min with Together AI

Dippy AI, an AI native startup providing AI companion apps, faced significant challenges scaling its infrastructure to meet rapid user growth. Their engineering team was spending excessive time managing inference infrastructure, which distracted from product development. They needed a solution that could handle cyclical traffic patterns, reduce latency, and improve cost efficiency. To address this, they partnered with Together AI to utilize their Dedicated Endpoints service.

By deploying their custom models on Together AI's Dedicated Endpoints powered by NVIDIA HGX H100 GPUs, Dippy AI achieved optimal performance. Together AI provided the necessary optimizations and auto-scaling infrastructure. The results were substantial: median time to first token was reduced to 0.4 seconds, peak throughput reached 4.1 million tokens per minute, and average latency dropped to 3.44 seconds. Most importantly, Together AI's solution allowed Dippy AI's team to fully focus on building user-facing features instead of managing infrastructure.

Dippy AI