Case Study: Notion deploys frontier AI models within hours with Braintrust

Notion deploys frontier AI models within hours with Braintrust

The customer, Notion, a connected workspace platform, faced the challenge of scaling its AI evaluation practices to keep 70 engineers aligned while rapidly deploying new frontier language models. Their previous methods were insufficient for measuring complex agentic AI and for identifying rare but critical problems in their vast and unique dataset. They turned to the vendor, Braintrust, and its AI evaluation platform to address this.

Braintrust provided the solution by enabling Notion to build a comprehensive evaluation framework with both regression and frontier evals. This allowed Notion's team to quickly identify model differences, find needle-in-a-haystack problems like multilingual errors, and deploy custom code for their specific data needs. The results were significant, enabling Notion to deploy new frontier models in less than 24 hours and achieve major quality improvements for key customer subsets, all while keeping their large engineering team aligned.

Notion