Case Study: a supply chain management platform improves deduplication accuracy and processes 2 billion records with Vinta Software

a supply chain management platform processes 2 billion records in hours with Vinta Software

A supply chain management platform needed to improve the accuracy and scale of its data enrichment process, specifically for deduplicating its massive datasets. The platform contacted Vinta Software after being impressed by a partner's talk on record linkage, seeking their expertise to handle over two billion records efficiently.

Vinta Software implemented a solution that enhanced the client's ETL pipeline, utilizing async.io and multiprocessing for parallelization and integrating scikit-learn to radically improve indexing speed. This allowed the platform to process two billion records in just a couple of hours with improved accuracy. Vinta also worked with the client's infrastructure team to establish new practices that freed them from time-consuming experiments, leading to increased satisfaction across other teams.