Case Study: Nova AI achieves 60x faster agent debugging with Patronus AI's Percival

Nova AI boosts SAP agent accuracy by 60% with Patronus AI

Nova AI, a company building an AI-powered platform for SAP custom code modernization, faced significant challenges in evaluating and debugging its complex, multi-agent workflows. Their agents, which take 20-30 minutes to run, required painstaking manual inspection that took about an hour per trace, creating a major bottleneck for improvement. To overcome this, Nova AI turned to vendor Patronus AI and its automated evaluation product, Percival, to systematically analyze agent errors and optimize performance.

By implementing Patronus AI's Percival, Nova AI gained an automated workflow to trace, score, and categorize agent failures. The solution provided actionable prompt fixes for domain-specific issues, such as validation of SAP object activation and annotation value length. This resulted in a dramatic 60x productivity boost by reducing debugging time from one hour to one minute. Furthermore, through rapid experimentation enabled by Patronus AI, Nova AI increased its agent's accuracy by 60% on an internal SAP tool dataset.

Nova AI