Fairness: Who Gets Left Out?

In the last lesson you learned that bias enters AI through data, design, and deployment. Now the question gets harder: what does it actually mean for an AI system to be fair? The word sounds simple. It is not. Researchers have identified over twenty mathematically distinct definitions of 'fairness' — and they often cannot all be satisfied at the same time. This is not a technical glitch; it is a genuine moral tension, and understanding it makes you a more careful thinker.

The Gap Between Performance and Impact

When an AI system performs differently across groups, the consequences are not abstract. They are concrete disadvantages for real people. In dermatology, AI systems trained to detect skin cancer have shown lower accuracy on darker skin tones. This is not a minor footnote: missed diagnoses or false alarms in medicine have direct health consequences. The gap in accuracy translates directly into a gap in quality of care. In natural language processing, voice recognition systems have historically performed worse on speakers with non-standard accents, older voices, and certain speech patterns. If a voice-activated medical device mishears a patient's symptom description, that error can cascade into a clinical decision. Representation in training data and representation in outcomes are deeply linked. Groups that were underrepresented in the data that trained a system tend to be the same groups that the system serves worst.

The COMPAS recidivism tool (Correctional Offender Management Profiling for Alternative Sanctions) was used by courts in the United States to help judges make bail and sentencing decisions. An investigation by ProPublica in 2016 found that the tool was roughly twice as likely to falsely flag Black defendants as high risk (predicting they would re-offend when they did not) compared to white defendants, and roughly twice as likely to falsely flag white defendants as low risk (predicting they would not re-offend when they did). The company that built COMPAS responded that the tool was 'fair' by a different metric: the proportion of people who scored 'high risk' and actually did re-offend was similar across groups. Both claims were mathematically accurate. They were measuring different things. This is the fairness problem: different definitions of fairness can point in opposite directions, and choosing between them is not a technical decision — it is a moral one.

Four Ways to Define Fairness

Here are four real definitions used in AI research and policy. Each captures something important. Each has limits. Demographic parity: The system should produce similar positive outcome rates across groups. If 30% of one group gets approved for a loan, roughly 30% of other groups should too. Critics point out this can ignore real differences in relevant qualifications. Equal accuracy: The system should be equally accurate — make equally few errors — across groups. This is what COMPAS's creators were claiming. Equal false positive rates: The system should be equally likely to wrongly flag an innocent person across groups. This is what ProPublica measured. Individual fairness: Similar individuals should receive similar outcomes, regardless of group membership. This is intuitive but requires defining what 'similar' means — which is itself contested. The unsettling mathematical result: you generally cannot satisfy equal accuracy AND equal false positive rates AND demographic parity simultaneously unless the groups have identical base rates. This means fairness trade-offs are real, not solvable with better engineering alone.