Case Study: monday Service achieves faster AI evaluation and production-quality monitoring with LangSmith

A LangChain Case Study

Preview of the Monday Service Case Study

Monday Service speeds eval feedback loops 8.7x with LangChain

Monday Service, an AI-native enterprise service management platform, faced the challenge of ensuring the quality and reliability of its new LangGraph-based AI service agents without slowing down development. Their challenge was to catch potential quality issues in these autonomous agents before they reached users, requiring a robust evaluation strategy from the very beginning of the development cycle. They partnered with vendor LangChain to address this using the LangSmith platform.

LangSmith provided a solution built on a two-pillar evaluation strategy. This included offline "safety net" evaluations run in parallel using the LangSmith Vitest integration for massive speed gains, and online multi-turn evaluations for real-time monitoring of production traffic. This approach resulted in an 8.7x faster evaluation feedback loop, reducing test time from 162 seconds to just 18 seconds, and enabled comprehensive testing across hundreds of examples in minutes. LangSmith's infrastructure allowed Monday Service to manage their evaluation logic as version-controlled code, ensuring high-quality, reliable agents.


View this case study…

LangChain

30 Case Studies