Bias Auditing for Production ML Systems

Bias auditing moved from research topic to regulatory expectation. EU AI Act, NY Local Law 144 for employment, CFPB guidance on model risk, state-by-state insurance regulations — the patchwork is real, and “we didn’t audit because we didn’t think we had to” is not a defense.

The four-axis audit we run on every production model.

Axis 1: outcome parity across protected attributes#

For each protected attribute (gender, race, age band, geography), measure model outcomes:

• Approval rate
• False positive rate
• False negative rate
• Calibration (do predicted probabilities match observed outcomes?)

Compare across groups. Significant disparities trigger investigation.

Two metrics matter most:

Demographic parity. Are approval/positive-outcome rates similar across groups? Useful as a high-level signal, easily misleading without context.

Equalized odds. Are false positive and false negative rates similar across groups, conditional on the true label? More technically defensible; harder to satisfy.

There’s no single right metric — there’s a set of metrics that surface different problems. Pick the ones that map to the decision being made.

Axis 2: counterfactual robustness#

For a sample of predictions, flip the protected attribute and see if the prediction changes. If it changes meaningfully on identical otherwise-data, the model is using the protected attribute (directly or via a proxy).

This catches the “we don’t include race as a feature, but ZIP code is a near-proxy” failure. Test the counterfactual; trust the test, not the feature list.

Axis 3: calibration analysis#

A model can be miscalibrated overall but appear unbiased on parity metrics. Or perfectly calibrated overall but miscalibrated within a subgroup. Plot reliability diagrams per subgroup.

Common pattern: a fraud model that predicts 30% fraud risk for one population segment whose actual fraud rate is 15%. Same predicted probability, very different precision in practice.

Axis 4: explanation quality#

For high-stakes models, run explanation tools (SHAP, integrated gradients, counterfactual explanations) on a stratified sample. The questions:

• Are the top features the same across subgroups?
• Do the explanations make domain sense?
• Are any features that should not be used appearing in the top contributors?

Explanations don’t prove fairness; they surface inconsistencies worth investigating.

When parity metrics conflict#

You will find that satisfying one fairness metric makes another worse. Demographic parity and equalized odds are in tension when base rates differ across groups. There is no purely-mathematical resolution; the choice depends on the decision context and the regulatory regime.

What works in practice:

• Document the metric you optimize for and why
• Document the metric you accept residual disparity on and what compensating controls exist
• Have the documentation reviewed by legal/compliance, not just data science

The audit cadence#

Pre-deployment. Full four-axis audit on the eval set. Block deploy on findings above tolerance.

Quarterly. Re-audit on production data. Catch drift.

On model change. Same audit, before promote.

On regulation change. Re-read the rules; update the audit if metrics changed.

Anti-patterns#

The “we removed the column” defense. Removing race from features doesn’t remove race-related signal. Proxies are everywhere. Audit, don’t assume.

Single-metric obsession. Reporting only demographic parity. The model that satisfies it might have terrible calibration for the affected groups.

Compliance-only thinking. Treating bias audit as a check-the-box exercise produces poor models and poor defense in court. Build it as part of model quality, not separate from it.

What we ship by default#

For ML engagements involving protected attributes or regulated industries via our AI & LLM integration service:

• Four-axis audit pre-deployment
• Quarterly re-audit baked into operations
• Calibrated probability outputs (Platt or isotonic)
• Documented metric choices and compensating controls
• Legal/compliance review of the audit framework, not just outputs

Bias audit is no longer optional. It also makes models genuinely better.

---

Bias audit is part of model quality, not separate from it. Our team ships audited, defensible ML systems across regulated industries. Tell us about the use case.