Models treat everyone the way their data treats them. If a hiring model’s examples come from ten years of biased hiring, it learns that bias perfectly and repeats it at scale. Nobody has to intend it: the bias was already in the data.
Three common sources of bias
- Skewed samples: some groups barely appear in the training data.
- Historical bias: the data faithfully records an unfair past.
- Proxy variables: the model uses zip code or name as a stand-in for something it shouldn’t.
Real stakes: loan approvals, medical triage, face recognition, resume screening. When an AI system makes these calls, an accuracy gap between groups becomes an opportunity gap.