Case Study: Avanade detects data drift and catches upstream data quality issues with Great Expectations

How Avanade uses GX to detect data drift from upstream model changes in machine learning pipelines

Avanade, a global professional services company, needed a way to spot frequent upstream data model and taxonomy changes that could silently break its machine learning pipelines. The Intelligent Enterprise Team was seeing data drift, zeroed-out features, and occasional outlier “dummy values” across data from systems like sales, HR, and Office 365, and wanted better visibility before these issues affected stakeholders.

Great Expectations (GX) was used to automatically profile data and build initial Expectation Suites, which the team then manually refined and applied throughout its Azure ML pipelines. They validated data at each pipeline step, generated HTML validation reports in Data Docs, and gained transparency into what changed and where; as a result, Avanade was able to catch upstream data quality problems before stakeholders noticed and reduce the risk of model degradation.

Avanade

Steve Nelson

Data Scientist