Day 60 of 60
·
AI-system specific
ML model & feature drift detection
Models decay silently as the world drifts away from training data. Without drift detection, the first to notice is Sales, looking at retention.
ProblemModel performance degrades silently as input distribution shifts away from training data.
How it works
Monitor input feature distributions, output distributions, and (where available) ground-truth performance. Alert on statistical drift. Trigger retraining or rollback.
What it catches
Distribution shift, label drift, concept drift, silent model decay. Without it, models degrade and only Sales notices when retention drops.
Tools
Evidently AI · OSS Arize · Hybrid WhyLabs · Hybrid Fiddler · SaaS
Verdict by project size
Small
Skip
Medium
Opt
Large
Rec
Extra-large
Must
Cost
| Project size | Setup | Maint / mo | Tool / mo | CI / run |
|---|---|---|---|---|
| Small <10k LOC | 1d | 1h | $0 | , |
| Medium 10–100k LOC | 3d | 5h | $300 | , |
| Large 100k–1M LOC | 15d | 25h | $3k | , |
| Extra-large >1M LOC | 60d | 120h | $25k | , |
Setup = engineer-days to first useful run ·
Maint = engineer-hours / month at steady state ·
Tool = out-of-pocket $ / month ·
CI = minutes added (or saved) per pipeline run
Lifecycle & ownership
When in lifecycle
Test Operate Observe
Per merge · Runs after merge to main; nightly heavy jobs.
Who owns it
ML / AI Engineer
Models, evals, drift, guardrails
Collaborates with: Developer, Security / AppSec
Reference implementations
-
Evidently examples
Model and data drift monitoring notebooks and reports.
-
WhyLabs examples
Data and model monitoring examples using whylogs profiles.
-
NannyML examples
Post-deployment model performance estimation and drift examples.
Quick check
Without ML drift detection, model decay is typically discovered by…
One question. Pick the best answer. Your streak is saved locally on this device.
Save the lesson
Download SVG ↓
Screenshot for a 1:1, drop it in Slack, or download the SVG.