Drift Detection: Statistical Tests vs Model-Based Approaches

A model trained in February degrades in October without anyone noticing — until a metric the business cares about quietly slides. Drift detection is the early-warning system. Done well, it catches degradation before customers do. Done badly, it produces alert fatigue that the team learns to ignore.

The two families of detection, where each fails, and what we deploy.

Family 1: statistical tests on input features#

Compare the distribution of a feature in production against its training distribution. If they differ significantly, flag.

Common tests:

• Kolmogorov-Smirnov for continuous features
• Chi-squared for categoricals
• Population Stability Index (PSI) — used heavily in credit risk
• KL or JS divergence between empirical distributions

The fundamental problem: most production models have dozens of features. With p < 0.05 on each, you’ll have multiple false alarms per day from chance alone. Bonferroni correction helps; pre-selecting the features you actually care about helps more.

The deeper problem: a feature can drift dramatically without harming predictions, and predictions can degrade without features drifting (the joint distribution moved). Feature-level tests miss this.

Family 2: model-based detection#

Train a binary classifier to distinguish training-period data from current data. If the classifier achieves >random AUC, the data has drifted.

This catches multivariate drift that univariate tests miss. The classifier’s important features tell you where drift is concentrated.

Variants:

• Domain classifier. Trained directly to distinguish reference vs current.
• Reconstruction error. Autoencoder trained on reference; reconstruction error on current samples flags drift.
• Density ratio estimation. Direct estimate of ratio without modeling either distribution explicitly.

Cost: more compute, more pipeline.

The label-drift problem#

Both families above watch inputs. They miss the most important kind of drift: the relationship between inputs and labels changing. A perfectly stable input distribution can still produce poor predictions if the underlying behavior changed.

Detecting label drift requires either:

• Real labels (which arrive with delay — sometimes substantial)
• Proxy labels (downstream business outcomes that correlate)
• Performance proxies (model confidence distribution, agreement with simpler baselines)

In our experience, the model’s confidence distribution over time is the most useful cheap signal. A confident model becoming less confident is suspicious even before labels arrive.

The hybrid we deploy#

For each production model:

• Input drift. PSI on top 10–20 features, with thresholds calibrated against historical natural variation. Alerts only on features the model actually weighs heavily.
• Joint drift. Model-based detection running weekly. AUC threshold tuned for the domain.
• Output drift. Distribution of model predictions over time. Sudden shifts in the predicted-probability histogram are diagnostic.
• Performance proxies. Confidence distribution, agreement with a simpler baseline, downstream business metric.
• Real performance when labels arrive. Backfilled into the dashboard so the team can see lagged truth.

Calibrating thresholds#

Threshold-by-vibes doesn’t work. The discipline:

• Hold out 6+ months of pre-deployment data
• Simulate drift detection on rolling windows of that data
• Tune thresholds so the false alarm rate matches your tolerable rate (often 1–2 per month)
• Re-tune after the first 3 months of production data

Without this calibration, alerts fire too often (and get ignored) or too rarely (and miss real drift).

When to retrain vs investigate vs ignore#

A drift alert is the start of a question, not an answer.

Investigate. Look at which features drifted. Map to known causes — seasonality, product changes, regulatory shifts, customer mix change.

Retrain. If the drift is sustained and impactful. Don’t retrain on every weekly alert; you’ll chase noise.

Accept. Some drift is benign — input distribution shifted but performance held. Document and move on.

What we ship by default#

For ML systems via our DevOps automation service:

• Input drift monitor on key features with calibrated thresholds
• Weekly model-based drift detection
• Confidence distribution dashboard
• Performance backfill once labels are available
• Documented investigate/retrain/accept decision flow

Drift detection earns its keep by buying you time to react before the business notices. Make sure the alerts are actionable, not just noisy.

---

An ML system without drift detection is a system that’s already drifting silently. Our team ships production-monitored ML systems. Tell us about the model.