Case Study: Meta achieves instant code visibility and live debugging at 24k-GPU scale with Hammerspace

Meta enables interactive debugging across 24,576-GPU clusters with Hammerspace

Meta, a hyperscale tech company focused on artificial intelligence, faced a significant challenge with interactive debugging at scale across its two massive 24,576-GPU clusters used for AI training. The difficulty of identifying a single problematic node stalling a training job hindered their development velocity. They partnered with Hammerspace to co-develop a solution for this problem.

Meta and Hammerspace built a parallel Network File System (NFS) deployment that was paired with Meta's "Tectonic" distributed storage. This Hammerspace solution provided instant data visibility, enabling engineers to debug jobs live and propagate code changes immediately. The result was fast iteration velocity that supported the training of the Llama 3 model across their 24k-GPU clusters without compromising on the exabyte scale required for data loading.

Meta