Gender Shades
In plain wordsA 2018 study showing that commercial face-analysis software guessed the gender of darker-skinned women far less accurately than that of lighter-skinned men.
An audit study by Joy Buolamwini and Timnit Gebru that tested commercial gender-classification services from Microsoft, IBM and Face++ on a new benchmark, the Pilot Parliaments Benchmark: 1,270 photographs of members of parliament from three African and three Nordic countries, labeled by gender and by Fitzpatrick skin type. Error rates were at most 0.8% for lighter-skinned men but reached 34.7% for darker-skinned women, and every system performed worst on that intersection. The study showed that aggregate accuracy can hide subgroup failures, popularized intersectional evaluation (reporting results for combinations of attributes) and demonstrated that public audits can change industry practice: the vendors released improved versions within months.
Gender Shades is why we report accuracy for every skin-type and gender subgroup, not just the headline number.
Origin
Grew out of Buolamwini's research at the MIT Media Lab and was published at the first Conference on Fairness, Accountability and Transparency (FAT*, February 2018). A follow-up, 'Actionable Auditing' (Inioluwa Deborah Raji and Buolamwini, 2019), found that the audited firms had narrowed their gaps, while newly audited Amazon and Kairos showed similar disparities.
Usage note
The study measured binary gender classification, not identification of individuals, and its gender labels were themselves binary, limits later critics and the authors acknowledged. It influenced the 2020 decisions of several firms to pause or end sales of facial recognition to police.
Related
- Joy BuolamwiniJoy Buolamwini, prop. n.: A Canadian-American computer scientist and activist whose research showed that face-analysis software works far worse on darker-skinned women, and who founded the Algorithmic Justice League.
- Timnit GebruTimnit Gebru, prop. n.: An Ethiopian-born American computer scientist who exposed racial and gender bias in AI systems, co-wrote the 'stochastic parrots' paper, and founded an independent AI research institute.
- facial recognitionfacial recognition, n.: Technology that identifies or checks who someone is by comparing a picture of their face with stored photos, used by phones, passport gates and police databases.
- algorithmic biasalgorithmic bias, n.: Unfairness in software that judges or serves people: it consistently produces worse results for some groups, such as women, older people or ethnic minorities, often because it learned from skewed or incomplete records.
- AI auditAI audit, n.: An independent check of an AI system, or of the company using it, against agreed standards — for example, testing whether a hiring tool treats women and men fairly.
- group fairnessgroup fairness, n.: Measuring fairness by comparing how a system treats whole groups, such as women and men, on average: their approval rates, their error rates or what their scores mean.
Sources
- Buolamwini, J. & Gebru, T. (2018). Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. Proceedings of Machine Learning Research 81 (FAT* 2018), 77–91. proceedings.mlr.press/v81/buolamwini18a.html
- Raji, I. D. & Buolamwini, J. (2019). Actionable Auditing: Investigating the Impact of Publicly Naming Biased Performance Results of Commercial AI Products. Proc. AIES 2019.
Mentioned in
algorithmic biasalgorithmic bias, n.: Unfairness in software that judges or serves people: it consistently produces worse results for some groups, such as women, older people or ethnic minorities, often because it learned from skewed or incomplete records., facial recognitionfacial recognition, n.: Technology that identifies or checks who someone is by comparing a picture of their face with stored photos, used by phones, passport gates and police databases., biasbias, n.: A word with several AI meanings: an adjustable number added inside a model; a systematic error in estimates; or unfair treatment of some groups of people by an AI system., Joy BuolamwiniJoy Buolamwini, prop. n.: A Canadian-American computer scientist and activist whose research showed that face-analysis software works far worse on darker-skinned women, and who founded the Algorithmic Justice League., sampling biassampling bias, n.: A distortion that arises when the data collected over-represents some kinds of people or situations and under-represents others, so conclusions drawn from it don't hold for everyone., Timnit GebruTimnit Gebru, prop. n.: An Ethiopian-born American computer scientist who exposed racial and gender bias in AI systems, co-wrote the 'stochastic parrots' paper, and founded an independent AI research institute., Algorithmic Justice LeagueAlgorithmic Justice League, prop. n.: Joy Buolamwini's organization, known for showing that face-analysis software made far more mistakes with darker-skinned women, which campaigns to hold the makers of biased AI systems accountable., FAccTFAccT, prop. n.: A yearly academic conference on the fairness, accountability and transparency of computer systems, where much influential research on AI bias and ethics has been published.