Case Study: Khan Academy strengthens K–12 AI safety evaluation with HumanSignal Data Services

Khan Academy speeds AI safety evaluation 2x with HumanSignal

Khan Academy, a global educational platform, needed to create a highly custom AI moderation system to ensure content safety for its K-12 learners. The challenge was to build a foundation of high-quality training data calibrated for young students, which required annotators with specific expertise in child development. They partnered with vendor HumanSignal to use their Data Services to address this challenge.

HumanSignal provided a specialized annotation service, employing K-12 educators and childcare professionals to classify and flag messages. The project was completed in two phases and achieved over 90% annotator agreement. The solution helped Khan Academy deliver its custom moderation system twice as fast as estimated with an internal team. The high-quality dataset provided by HumanSignal now serves as a critical evidence base for ongoing AI safety evaluations, benefiting millions of learners.

Khan Academy