Case Study: Northeastern University improves agentic AI evaluation with Prolific

Northeastern University compares 40 users with digital twins using Prolific, finding 89% action-path divergence

Northeastern University, led by Dr. Dakuo Wang, faced a significant challenge in evaluating new agentic AI shopping systems. These systems, like Amazon's Rufus, are non-deterministic and personalized, making them impossible to assess with traditional methods that take months. By the time a formal study was complete, the AI would have already been updated multiple times, rendering the evaluation obsolete.

Prolific provided the solution by supplying 40 high-quality, demographically diverse human participants for a foundational study. The research used these participants to create "digital twin" AI agents and then compared the behavior of the real people to their digital counterparts. The results showed that while digital twins are excellent for fast, cheap testing of task completion, only real humans can provide the critical feedback on frustration, trust, and subjective experience that determines a product's real-world success. Prolific's role was essential in the crucial human-in-the-loop phase of a new hybrid evaluation model advocated by the research.

Northeastern University