Case Study: Dropbox achieves real-time AI search regression detection with Braintrust

Dropbox builds AI search evaluation pipeline with Braintrust and 10,000+ tests

Dropbox, a company providing file hosting and collaboration tools for over 700 million users, faced challenges in evaluating the AI search functionality for its new product, Dash. Their initial manual process using public datasets and spreadsheets was inefficient and could not scale, leading to a whack-a-mole situation with prompt changes and regressions. To solve this, they turned to the vendor Braintrust for its enterprise-grade AI evaluation platform.

With Braintrust, Dropbox implemented a comprehensive, multi-tier evaluation pipeline. This included pre-merge smoke tests, a full post-merge suite of over 10,000 tests, and online LLM-as-a-judge scoring in production for real-time regression detection. The solution provided by Braintrust enabled a continuous feedback loop from production data, allowing the team to ship code with confidence and systematically improve quality based on real-world usage patterns.

Dropbox