Case Study: LinkedIn achieves order-of-magnitude speedups in extreme-scale optimization with PyTorch

LinkedIn achieves 75x faster optimization with PyTorch

LinkedIn, the professional networking platform, faced a significant challenge in scaling its optimization systems to handle extreme-scale linear programming problems. These problems, involving trillions of decision variables, were used for tasks like matching jobs to seekers and optimizing email volumes. Their existing CPU-bound solver, DuaLip, built on a Scala/Spark stack, was struggling with the computational demands and was difficult to extend for new use cases. They needed a solution that could deliver faster performance and greater flexibility.

To address this, LinkedIn re-architected its DuaLip solver using PyTorch to create a new GPU-accelerated system called DuaLip-PyTorch. By leveraging PyTorch's sparse tensor operations and efficient matrix-vector computations, they transformed the solver's core operations to run on GPUs. This solution from PyTorch achieved order-of-magnitude speedups, being 75 times faster per iteration, and enabled efficient scaling across multiple GPUs. The result was production-grade optimization at a previously infeasible scale, significantly reduced engineering overhead, and a unified stack for both machine learning and optimization tasks.

LinkedIn