What bias testing can and cannot detect
Bias testing cannot tell you whether a system is fair. It can tell you whether a specific statistical property holds across groups you were able to identify, on data you happened to have, for outcomes someone recorded. That is a real and useful thing, and it is much narrower than the phrase implies. The most consequential biases in deployed systems are usually in the label, in who never appears in the data, and in how humans use the output, and none of those show up in a standard fairness report.
What can bias testing not detect?
Four things, and they are the ones that most often matter. Bias in the label, where the outcome the model learned to predict was itself the product of a biased process. A model trained to predict which employees get promoted learns the promotion behaviour of the organisation, including whatever was wrong with it, and it will pass every fairness test that compares its predictions against historical promotions because the historical promotions are the yardstick.
Second, bias against people absent from the data. If a group rarely applied, was systematically screened out earlier in the funnel, or is small enough that subgroup estimates are noise, testing returns a reassuring result about a population you never sampled. Intersectional subgroups fail this way routinely: a system can pass on sex and pass on ethnicity and fail badly for a specific combination that has too few records to measure.
Third, bias in how the output is used. Two teams handed the same score will override it at different rates for different applicants, which is a real disparity created after the model and invisible to any test of the model. Fourth, representational harm in generative output, where there is no outcome variable to compare and the harm is in what the system produces rather than in a decision it makes. Testing helps with none of these, and a governance process that treats a passed bias test as the end of the fairness question has mistaken the map for the territory.
Which fairness metric should you use?
Whichever one matches the harm you are trying to prevent, and you must choose, because you cannot have them all. This is not a resourcing problem, it is a proven mathematical result: when base rates genuinely differ between groups, a score cannot simultaneously be calibrated within each group and produce equal false positive and false negative rates across them, except in degenerate cases. Any vendor claiming a model satisfies every fairness definition is describing something that cannot exist.
The choice therefore turns on which error costs more and to whom. If a false negative denies someone an opportunity, equal opportunity is the relevant property. If a false positive subjects someone to an intervention they did not deserve, false positive parity matters more. If a downstream decision-maker will read the score as a probability, calibration matters most, and if it is not calibrated within each group they will systematically misread it for some people.
| Metric | What it equalises | Choose it when | What it costs |
|---|---|---|---|
| Demographic parity | Selection rate across groups | Access itself is the outcome and base rates are not legitimately different | Forces different thresholds and can reject better-qualified candidates |
| Equal opportunity | True positive rate across groups | Missing a qualified person is the harm you care about | Says nothing about false positives, which may fall unevenly |
| Equalised odds | True and false positive rates across groups | Both error directions carry real cost | Incompatible with calibration when base rates differ |
| Calibration within groups | Meaning of a given score across groups | A human reads the score as a probability and acts on it | Incompatible with equal error rates when base rates differ |
| Impact ratio (four-fifths rule) | Ratio of selection rates against the highest group | Employment screening, where it still governs private Title VII litigation and New York City Local Law 144 audits, though federal enforcement has retreated from it since 2025 | A coarse screen, not a fairness proof, and easy to satisfy while harming people |
| Counterfactual or individual fairness | Outcome when only the protected attribute changes | You need to test a specific causal claim about a decision | Requires a causal model most teams cannot justify |
What do you need before you can test at all?
Three things, and the second is where most programmes stop. You need outcomes recorded, you need the protected attribute for each record, and you need enough records in each group for the estimate to mean anything. Without the attribute there is nothing to disaggregate by, and most organisations either do not collect ethnicity and disability data or collect it for a purpose that does not permit this use.
That produces a familiar deadlock: data protection considerations discourage holding the attribute, and fairness testing is impossible without it. The resolution in European practice is usually that testing for discrimination is a legitimate purpose that can justify processing special category data under an appropriate condition, handled with strict access separation, but it needs a deliberate legal basis rather than an assumption. Establish it before the model is built, not after it is live.
The sample size point is less discussed and more often fatal. Estimating a selection rate for a group with forty records gives a figure with a confidence interval wide enough to accommodate almost any conclusion. Report intervals rather than point estimates, and when a subgroup is too small to measure, record that as an unmeasured group rather than letting it disappear into an aggregate that looks fine.
What can you do without demographic data?
Less than a full analysis, but more than nothing, and it is worth being clear about which techniques are defensible. Proxy inference from names and postcodes exists, is used in some regulated contexts, and produces group-level estimates rather than individual ones. It is acceptable for measuring aggregate disparity in some jurisdictions and firmly not acceptable as a basis for treating an individual differently. Treat it as a monitoring instrument only, and document its error characteristics, because the inference is wrong for a meaningful share of people.
Two alternatives avoid the attribute altogether. Voluntary, separated collection at the point of application, held apart from the decision system and used only for aggregate reporting, is the cleaner route where the volume supports it. And a paired testing approach, where you construct matched synthetic applications differing only in the signal you are worried about, tests the model directly without needing any real person's attributes at all.
Paired testing is the one most teams can run this week. Take fifty real inputs, produce variants that differ only in a name, a postcode, a gender marker or a career gap, and compare the outputs. It does not prove fairness across your population, but it detects the crudest and most embarrassing failures cheaply, and it is reproducible evidence that someone looked.
What does a defensible bias test look like?
It states the decision being tested, the population it was tested on and how that population relates to the deployed one, the metric chosen and the reason for choosing it over the alternatives, the results with confidence intervals and by subgroup including intersections where sample size allows, the subgroups too small to measure, and the date and model version.
It also states the comparison. A model that shows a disparity is not automatically worse than what it replaced, and a report that omits the incumbent process leaves the most important question unanswered. Where the previous process was human and unmeasured, say so, and say what you did to establish a baseline rather than implying there was none.
Some jurisdictions have made a version of this mandatory and published. Employment screening tools used in New York City require an annual independent bias audit computing selection or scoring rates and impact ratios three ways: by sex category, by race and ethnicity category, and by intersectional category combining sex with race and ethnicity. A summary of the results must be published, and candidates must be notified at least ten business days before the tool is used on them. The intersectional requirement is the demanding part, and it is the part that matters most, because a system that passes on sex and passes on ethnicity can still fail badly for a specific combination. Whether or not you are in scope, that specification is a reasonable floor for what a credible test contains, because it was written to be checkable by an outsider.
What do you do when you find a disparity?
Start by finding out where it came from, because the remedy differs entirely depending on the source and applying the wrong one usually makes things worse. A disparity from unrepresentative training data is fixed with data. A disparity from a feature acting as a proxy for a protected attribute is fixed by examining the feature, though removing it blindly often fails because the information survives in correlated features. A disparity from a threshold set once for the whole population may be a threshold problem. A disparity from a biased label is not fixable within the model at all, and the honest conclusion is sometimes that the target variable was the wrong thing to predict.
Be careful with post-hoc adjustments that equalise a metric by group. Group-specific thresholds change the decision made about individuals on the basis of a protected characteristic, which in several jurisdictions is unlawful regardless of the intent, and in employment contexts particularly so. The technique is statistically straightforward and legally fraught, and that ordering surprises engineering teams.
The decision that most often gets deferred and should not is whether to deploy at all. If a system shows a material disparity you cannot explain, running it while you investigate means the harm accrues to real people during the investigation. Not deploying is an available option, and a governance process where it has never once been chosen is not exercising judgement.
Common questions
- Can you test an AI model for bias?
- You can test whether a specific statistical property holds across groups you can identify, on data you hold, for outcomes that were recorded. That is useful but narrow. It cannot detect bias in the label the model learned from, harm to people absent from the data, disparity created by how humans use the output, or representational harm in generative output where there is no outcome variable to compare.
- Why can't a model satisfy all fairness metrics at once?
- Because it is mathematically impossible when base rates genuinely differ between groups. A score cannot be simultaneously calibrated within each group and produce equal false positive and false negative rates across groups, other than in degenerate cases. This is a proven result rather than an engineering limitation, so the choice of metric is a judgement about which error costs more and to whom, and it has to be made and documented.
- What is the four-fifths rule?
- A screening standard from US employment selection guidance: if the selection rate for one group is less than eighty per cent of the rate for the highest-selected group, that is generally treated as evidence of adverse impact warranting investigation. It is a coarse trigger rather than a fairness proof. A system can satisfy it while still harming individuals, and can fail it for reasons that turn out to be legitimate on examination. Its federal standing has narrowed: Executive Order 14281 in April 2025 directed agencies to deprioritise disparate impact enforcement, a Department of Justice opinion of 9 June 2026 concluded the Uniform Guidelines on Employee Selection Procedures are unlawful, the Office of Personnel Management removed them from federal personnel regulations by interim final rule in August 2026, and the Equal Employment Opportunity Commission has a pending rulemaking to rescind the related recordkeeping requirements. The four-fifths text remains in the Code of Federal Regulations at 29 CFR 1607.4(D), and the rule remains operative in private Title VII litigation and in New York City Local Law 144 audits, so treat it as a live standard in those forums rather than as a settled federal reference point.
- How do you test for bias without collecting demographic data?
- Three routes. Voluntary collection separated from the decision system and used only for aggregate reporting is the cleanest. Proxy inference from names and postcodes gives group-level estimates and is defensible for monitoring but not for treating any individual differently. Paired testing, where matched inputs differ only in the signal of concern, tests the model directly without needing any real person's attributes and can be run in a day.
- What is label bias in machine learning?
- When the outcome a model was trained to predict was itself produced by a biased process, so the model reproduces that bias while appearing accurate. A model trained on historical promotion decisions learns the organisation's promotion behaviour including its flaws, and will score well against historical outcomes because those outcomes are the yardstick. No amount of testing against the same labels reveals the problem, and it cannot be fixed inside the model.
- Should you use different thresholds for different demographic groups?
- It equalises metrics easily and carries serious legal risk. Setting a different decision threshold by protected characteristic means treating individuals differently on the basis of that characteristic, which is unlawful in several jurisdictions regardless of the intention behind it, and employment contexts are among the strictest. Investigate the source of the disparity first, because remedies differ depending on whether it originates in the data, a proxy feature, the threshold or the label.
