Where Bias Hides in Data

A model trained on biased data will produce biased predictions — not because anyone intended it, but because the data quietly encoded a distorted picture of the world. This is not a minor technical glitch. Biased ML models have wrongly flagged loan applications, led to unfair hiring decisions, and produced face recognition systems that work well for some people and poorly for others. Understanding where bias comes from is a professional responsibility for anyone who works with data.

Three Major Sources of Data Bias

Sampling bias happens when the examples in your dataset are not representative of the people or situations you want the model to work for. A classic historical example: in the 1930s, a magazine ran a presidential poll and predicted a landslide for the wrong candidate. They had surveyed millions of people — but almost all were wealthy car and telephone owners, not the broader voting population. Their massive sample was massively biased. In ML today, sampling bias appears when a medical dataset is collected at one hospital in one city — and then a model trained on it is deployed across the whole country. Patients in rural areas, or patients from different ethnic backgrounds, may have systematically different health profiles that the training data never captured.

Historical bias is more insidious. It occurs when the training data accurately reflects the past — but the past contained unfair patterns that should not be reinforced. A hiring algorithm trained on ten years of a company's hiring decisions will learn that the company historically hired mostly men for engineering roles. The data is not collected incorrectly — those hires really happened. But the pattern reflects past discrimination, not an objective measure of who can do the job. A model that learns this pattern will perpetuate the discrimination automatically. Historical bias is difficult because the data is technically accurate. The problem is not how it was collected; it is what it records.

Label bias occurs during the annotation process — when the humans applying labels bring inconsistent or prejudiced judgments. If a dataset of essays is labeled for quality by annotators who unconsciously rate essays about certain topics lower, that bias flows into the model. If a medical dataset labels patients as 'high risk' based on how much healthcare they have historically used — and poorer patients use less care because of cost barriers — then 'high risk' actually means 'used a lot of healthcare,' not 'is actually sicker.' A real published study found exactly this pattern in a major healthcare algorithm used across the US.

How Bias Harms and What to Do

Bias in data harms real people. Hiring algorithms filter out qualified candidates. Credit algorithms deny loans to creditworthy applicants. Face recognition systems misidentify people of certain races at higher rates, which has led to wrongful arrests. Detecting and reducing bias requires deliberate action: Audit who is represented in your dataset and who is not. Compare model performance separately for different demographic groups — if accuracy is high for one group and low for another, bias is likely present. Question labels critically: what human judgment produced them, and could that judgment be biased? Do not assume that a model is fair just because it was trained on real data. Real data can encode real injustice.