A bias audit sounds like a technical exercise and is mostly a sequence of judgement calls with arithmetic in between. The arithmetic is straightforward: count outcomes by group, divide, compare. The judgement calls decide what the number means. Which groups. Which outcome counts as favourable. Which definition of fairness. What threshold. What to do when two defensible definitions disagree, which they routinely do.
This article is about doing it properly and knowing where the discretion sits. It is not about adversarial testing, which asks whether a system can be made to misbehave. That is a different discipline with different methods, covered in AI red-teaming and model testing. A model can be robust against attack and still distribute outcomes unequally, and it can be fair on every metric you measured and trivially jailbroken.
Three kinds of bias, and only one is arithmetic
NIST SP 1270, Towards a Standard for Identifying and Managing Bias in Artificial Intelligence, is the most useful framing available because it refuses to reduce the problem to a metric. It identifies three categories:
| Category | Where it comes from | Can measurement find it? |
|---|---|---|
| Systemic | Historical and institutional patterns encoded in the data and in the process the model automates | Only partly. If the historical outcomes were unequal and you train to predict them, the model reproduces the pattern faithfully and your metrics may look fine |
| Statistical and computational | Unrepresentative samples, systematic measurement error, modelling choices. Arises without prejudice or intent | Yes. This is the part a bias audit measures well |
| Human | How people specify the problem, label the data, interpret the output, and defer to it | Rarely by metric. Found by examining the process and the interface |
SP 1270 also names three mitigation challenges, datasets, testing and evaluation, and human factors, and a three-stage approach across pre-design, design and development, and deployment. The practical consequence is that an audit confined to output statistics inspects one category of three. That is worth doing and is not sufficient, and reporting it as though it covered the field is the most common overclaim in this area.
The systemic row is the uncomfortable one. If the historical decisions you trained on were themselves unequal, a model that predicts them accurately is working as designed and causing harm. No output metric distinguishes a model that is wrong from a model that is faithfully reproducing an unjust process.
Choosing a metric is choosing a policy
The common fairness definitions are incompatible. Except in degenerate cases you cannot satisfy them all at once, which means selecting one is selecting a position on what fairness means for this decision.
| Definition | Requires | Reasonable when |
|---|---|---|
| Demographic parity | Similar favourable rates across groups | You believe underlying qualification rates are similar, or you are targeting representation in the outcome |
| Equal opportunity | Similar true positive rates across groups | Missing a qualified person is the harm you care most about |
| Equalised odds | Similar true positive and false positive rates | Both error directions carry real cost, as in many risk-scoring settings |
| Calibration within groups | A given score means the same probability in each group | The score is handed to a human who will interpret it as a probability |
| Counterfactual fairness | The decision would be unchanged had the protected attribute differed | Conceptually attractive, demanding in practice because it requires a causal model |
Write down which definition you chose and why, before you measure. Choosing afterwards, once you can see which metric your model passes, is the most common way a bias audit becomes a marketing exercise. Where a regime prescribes the measure, as New York City does, use the prescribed one for compliance and any others you find informative for your own understanding.
Selection rate and impact ratio, worked
The measure most regulation actually asks for is a comparison of selection rates. New York City's Local Law 144 requires a bias audit of automated employment decision tools that calculates the selection rate for each category and the impact ratio, being the ratio of each group's rate to the rate of the group with the highest rate, across race, ethnicity and sex per EEOC component 1 categories, including intersectional combinations. The city's Department of Consumer and Worker Protection publishes the requirements.
Take a screening tool applied to 1,000 applicants.
| Group | Applicants | Advanced | Selection rate | Impact ratio |
|---|---|---|---|---|
| Group A | 500 | 150 | 30.0% | 1.00 (highest) |
| Group B | 300 | 66 | 22.0% | 0.73 |
| Group C | 150 | 39 | 26.0% | 0.87 |
| Group D | 50 | 11 | 22.0% | 0.73 |
Group A has the highest rate at 30 percent, so it is the reference. Group B's impact ratio is 22.0 divided by 30.0, which is 0.73. Two groups fall below 0.8.
Now the parts that require judgement rather than division. Group D has 50 applicants and 11 advanced. One more or one fewer moves its rate by two percentage points and its ratio by about 0.07. Reporting 0.73 for Group D as a finding of comparable weight to Group B's 0.73 overstates what the data supports. Report the counts alongside the ratios and say plainly where the numbers are thin. And note what this table does not tell you: whether the people advanced were the right people. A ratio can be 1.00 while the tool selects poorly in every group.
The four-fifths convention, and what it is not
The 0.8 threshold comes from the EEOC's Uniform Guidelines on Employee Selection Procedures, codified at 29 CFR Part 1607. It is a rule of thumb for when a difference in selection rates is generally regarded as evidence of adverse impact warranting scrutiny.
Three things it is not. It is not a safe harbour: a ratio above 0.8 does not establish that a practice is lawful, and statistically significant disparities can matter below that threshold. It is not a statistical test, which is why a small sample can produce an alarming ratio that means very little and a very large sample can produce a modest ratio that is highly significant. And it is not what Local Law 144 requires you to achieve. That law requires the ratio be calculated and published, not that it clear a particular value.
Use it as a trigger for investigation rather than a grade. Pair it with a significance test and with the counts, and be prepared to explain a ratio rather than defend a number.
Protected attributes you do not hold
The practical obstacle in most audits is that you cannot measure disparity across a characteristic you never collected, and often had good reason not to collect. Four routes, each with a genuine cost:
| Route | Trade-off |
|---|---|
| Voluntary self-identification | Most defensible and usually incomplete. Response rates differ by group, which biases the measurement itself |
| Separate the data | Hold attributes in a restricted store used only for audit, never for the model. Strong practice, requires real access control |
| Statistical inference from proxies | Creates a sensitive dataset you did not have and introduces its own error. Rarely worth it, and in some jurisdictions creates its own legal exposure |
| Aggregate or third-party audit | An auditor computes disparities without returning individual attributes to you. Increasingly the practical answer for regulated audits |
Proxy variables deserve separate attention, because they are how a model discriminates without the attribute. Postcode encodes ethnicity in many countries. Employment gaps correlate with caring responsibilities and so with sex. Institution attended encodes socioeconomic background. Testing whether protected attributes are predictable from your feature set is a useful diagnostic: if a model can infer them from your features, so can your main model, whatever you excluded from the input.
Intersectionality and honest sample sizes
Local Law 144 expects intersectional categories, and this is right in principle and awkward in practice. Disparity can be invisible in the marginal breakdowns and clear at an intersection. It is also where cell counts collapse fastest: a population comfortable for single-attribute analysis can leave a handful of people in an intersectional cell.
The defensible approach is to compute what the data supports, report confidence intervals rather than point estimates alone, and state explicitly where a cell is too small for a conclusion instead of quietly omitting it or presenting the number as though it were solid. A cell of nine people with a startling ratio is a prompt to collect more data, not a finding.
What independence requires
Local Law 144 requires the audit be performed by an independent auditor, and notably the DCWP does not maintain a list of approved auditors, so selection and defensibility are the employer's responsibility. In practice independence means the auditor is not employed by, and has not provided services to, the employer or the tool vendor in a way that creates a conflict; has no financial interest in the tool; and has relevant expertise, typically in industrial and organisational psychology, statistics, data science, or employment discrimination analysis.
Two failure patterns to avoid. The vendor's own audit of its own tool, offered to every customer, is not an independent audit of your use of it, and your selection rates depend on your applicant pool. And an internal audit by the team that built or bought the tool has the same structural problem as second-line staff writing first-line artefacts: nobody credibly challenges their own work. If it must be internal, place it in a function with no stake in the outcome and expect that to be questioned.
Where this is mandatory today
| Regime | Requires | Status |
|---|---|---|
| NYC Local Law 144 | Independent bias audit of automated employment decision tools, published selection rates and impact ratios, candidate notice | In force |
| EU AI Act | Data governance and bias examination duties for high-risk systems, plus fundamental rights assessment for certain deployers | Phased, with the high-risk regime applying from August 2026 |
| EEOC and Title VII | Existing anti-discrimination law applies to selection procedures whether or not a model is involved | Long-standing |
| Colorado | Disclosure, adverse-outcome explanation, data correction and human review. Not impact assessments | Duties from 1 January 2027 |
The Colorado row is worth being precise about, because a great deal of published commentary describes a regime that never commenced. Senate Bill 24-205 in 2024 created a risk-based framework with a duty of reasonable care against algorithmic discrimination, impact assessments and risk management programmes. Its start date was postponed from February 2026 to 30 June 2026, and then Senate Bill 26-189, signed on 14 May 2026, repealed and reenacted those provisions as something different: a disclosure, documentation and consumer rights regime built around automated decision-making technology, with duties beginning 1 January 2027 and Attorney General rulemaking under way. If you are planning against Colorado, plan against notice, a 30-day adverse-outcome explanation, data access and correction, and meaningful human review, and check the current entry in our policies and frameworks tracker before relying on any summary including this one.
Remediation
Finding a disparity is the easy half. Options, roughly in order of how well they hold up:
- Fix the data. Address unrepresentative sampling or systematic measurement error at source. Slowest and most durable.
- Reconsider the target variable. Frequently the real fault. If you predict a proxy for the outcome you care about, and the proxy is unequally recorded, the model inherits that. This is the highest-yield intervention and the least often attempted.
- Remove or re-engineer features. Effective where a specific proxy is doing the work, and often less effective than expected because the signal is distributed across correlated features.
- Adjust thresholds. Technically simple, legally sensitive in employment contexts. Take advice before using group-specific thresholds.
- Constrain during training. Fairness-constrained optimisation works, at some cost to overall accuracy, and requires you to have chosen a definition.
- Change the decision, not the model. Add human review at the margin, or narrow what the model is used for. Often the fastest route to reducing harm.
Whatever you do, re-test and record the before and after. An audit that documents a disparity and a remediation, with the post-remediation measurement, is a stronger artefact than an audit that found nothing.
Publish and retain
Where publication is required, as in New York City, it is a compliance artefact with a date and a location. Where it is not, the record still matters: what was tested, on what data, over what period, by whom, with which metric and why, the results with counts and intervals, the remediation, and the re-test. Retain it for as long as the applicable rules require, and treat the audit as recurring rather than annual by calendar. A retrain, a data source change, a threshold adjustment or a shift in the applicant pool all invalidate a prior result, which is the same logic that makes continuous monitoring the right frame for fairness as well as for performance.
Frequently Asked Questions
What is the difference between a bias audit and red-teaming?
A bias audit measures how outcomes are distributed across groups under normal operation. Red-teaming tries to make a system behave badly by attacking it. The methods, skills and outputs differ: an audit produces disparity statistics, a red-team exercise produces findings with severities. They also catch different failures, and neither substitutes for the other. A system can distribute outcomes evenly and be trivially jailbroken, or be highly robust and quietly disadvantage a group.
Does passing the four-fifths rule mean we are compliant?
No, on two counts. The four-fifths convention in the EEOC's uniform guidelines is a rule of thumb for when a difference warrants scrutiny, not a legal safe harbour, and statistically significant disparities below that threshold can still matter. Separately, New York City's Local Law 144 requires that impact ratios be calculated and published, not that they reach any particular value. Treat 0.8 as the point at which you owe yourself an explanation, not as a pass mark.
How do we audit for bias when we do not collect protected attributes?
The strongest options are voluntary self-identification held in a store separated from the model's inputs, or an audit arranged so that a third party computes disparities without returning individual attributes to you. Both are increasingly standard for regulated audits. Inferring attributes from proxies is a last resort: it creates sensitive data you did not previously hold, introduces its own error, and can create legal exposure of its own. Whichever route you take, document it, because the measurement method is part of the finding.
Can our vendor's bias audit satisfy Local Law 144 for us?
Generally not on its own, for a structural reason as much as a legal one: selection rates depend on your applicant pool, so a vendor audit run on other populations does not describe your use of the tool. The law also requires independence, and an audit by the tool's maker sits awkwardly against that. Vendor evidence is useful input and may reduce your work, but expect to need an audit of your own data by an auditor without a stake in the tool.
How often should a bias audit be repeated?
Annually where a regime specifies it, and on trigger otherwise, because a result describes one model version applied to one population over one period. Retraining, a change in a data source, a threshold adjustment and a shift in the applicant pool all invalidate a prior result. The stronger pattern is continuous measurement of disparity as a monitored metric, with formal audits at intervals, rather than treating fairness as an annual event.
What if two fairness metrics disagree?
They will, and that is a mathematical property of the definitions rather than a flaw in your analysis. Except where base rates are equal across groups, satisfying demographic parity and calibration simultaneously is not possible. The response is to decide which harm matters most in this decision, state that choice and its reasoning in the audit, and report the other metrics anyway so a reader can see the trade you made. Concealing the disagreement is what turns an audit into advocacy.
