Case Study: Ozon boosts search relevance and trains ranking models with Toloka

How Toloka Helps Rate the Quality of Search Results in the Ozon online store

Ozon, Russia’s leading multi-category e-commerce platform, needed high-quality reference samples to evaluate a new search engine, choose the best ranking model, and improve its search algorithm with machine learning. Building an internal labeling tool and assessor pool would take too long, so Ozon used the Toloka crowdsourcing platform to create labeled data and rapid evaluations of search results.

Toloka delivered a multi-stage crowdsourcing workflow — iterative training, a control set (60% threshold) and a main task (80% threshold), with each query rated by five tolokers and a recreated store interface in an iframe. In the test run Toloka handled 350 tasks in 40 minutes for $12 (147 performers joined, 77 completed training, 12 gained the skill); the main launch produced 40,000 tasks in a month for $1,150 (1,117 tolokers joined, 18 trained, 6 active on the largest pool). Using Toloka gave Ozon faster, more granular manual labels for top queries, helped identify search problems, established evaluation criteria, and provided scalable data for training and improving its search models.

OZON