Case Study: Cisco achieves actionable NLP-bot analytics with Mixpanel

How Cisco uses Mixpanel to measure bots

Cisco data scientist Irzana Golding explains how Cisco tackled a common analytics problem: bots are hard to evaluate because simple volume metrics (like message count or “chattiness”) can be misleading, and teams need measures of effectiveness — e.g., whether a bot helps users complete tasks or is frequently misunderstood. Cisco wanted a practical, testable way to visualize bot behavior and surface meaningful signals such as misrecognition and task completion.

The team mapped API.AI intents to a single Mixpanel event ("Intent Fired") and pushed intent names and parsed entities as event properties, then used distinct_id-based funnels, retention, and segmentation to measure conversion, misrecognition rate, first-time failures, and feature-driven return behavior. With only a few lines of code and Mixpanel dashboards, they gained immediate, actionable behavioral insights to quantify bot effectiveness, spot failures (like fallback intents), and prioritize improvements.

Cisco

Irzana Golding

Data Scientist