Case Study: Microsoft achieves improved Bing search quality and rapid market expansion with Appen

We helped the Bing team expand by rapidly ramping up new markets

Microsoft enlisted Appen to improve Bing search quality across the U.S. and international markets. The challenge was to produce large-scale, culturally relevant evaluated datasets of query–result pairs so Bing could benchmark and improve result relevance, accuracy and spam resistance; Microsoft needed rapid ramp-up of linguistic evaluators, reliable data collection and annotation, and scalable processes to enter new markets quickly.

Appen assembled and trained linguistic teams within weeks, hired and trained dozens of annotators in two months, developed a proprietary data analysis and reporting tool, and now processes millions of pieces of search data every month in more than a dozen markets. As a result, Microsoft could rapidly benchmark performance, expand into new regions, and continuously improve Bing’s search quality through ongoing training, program management, and iterative guideline feedback from Appen.

Microsoft