Case Study: Wizard cuts evaluation costs by 75% with SuperAnnotate and NVIDIA Nemotron

Wizard cuts evaluation costs by 75% with SuperAnnotate

Wizard, an AI-powered shopping agent, faced a significant challenge with its manual evaluation process for product recommendations. As query volume grew, the high cost and slow speed of using a team of 18 human annotators to review every single query became a bottleneck for scaling. The company partnered with SuperAnnotate to implement a new, scalable solution without sacrificing the high accuracy required for customer trust.

In collaboration with SuperAnnotate and NVIDIA, Wizard built a hybrid evaluation system using an NVIDIA Nemotron LLM Judge. The solution automatically handles high-confidence cases and routes only ambiguous ones for human review. This approach, implemented by SuperAnnotate, resulted in a 75% reduction in human annotation costs while maintaining a 96% residual accuracy rate and a 91% alignment with human expert judgment.

Wizard