Case Study: Hugging Face scales open-source model discovery with Langfuse

Hugging Face improves model discovery reliability with Langfuse after 2 rounds of error analysis

Hugging Face, a leading platform for open-source AI models, faced the challenge of efficiently discovering and onboarding the many new models published daily on arXiv but not shared on their Hub. Their Machine Learning Engineer, Niels Rogge, built an agent to automate this process but needed better observability to debug and trust its decisions, leading them to implement the Langfuse observability platform.

By integrating the Langfuse Python SDK, the team gained full visibility into each agent run, grouped by paper. This allowed them to identify and fix key issues, such as an unreliable classifier and missing context from GitHub READMEs. After two rounds of error analysis guided by Langfuse traces, the agent achieved high reliability, receiving only two negative comments across thousands of automated issues and pull requests, effectively scaling this outreach effort.

Hugging Face