algorithmic fairness
In plain wordsThe effort to make automated decisions fair: to say in numbers what fair treatment means, measure whether a system achieves it, and correct the system if it does not.
The research field, and the property it studies, concerned with formal criteria for non-discrimination in automated decisions and methods to satisfy them. Group criteriagroup fairness, n.: Measuring fairness by comparing how a system treats whole groups, such as women and men, on average: their approval rates, their error rates or what their scores mean. compare statistics across groups defined by a protected attributeprotected attribute, n.: A personal trait, such as race, sex, religion, age or disability, that the law says must not be used to treat people unfairly.: equal selection rates (demographic paritydemographic parity, n.: A fairness test that asks whether a system says yes (approves the loan, shortlists the applicant) at the same rate for every group, whatever other differences exist between them.), equal error rates (equalized oddsequalized odds, n.: A fairness test that asks whether a system makes mistakes at the same rates for every group: wrongly flagging innocent people, and wrongly missing real cases, equally often., equal opportunityequal opportunity, n.: A fairness test that asks whether people who truly deserve a yes (who would repay the loan or succeed in the job) get one at the same rate in every group.) or equally meaningful scores (predictive paritypredictive parity, n.: A fairness test that asks whether a system's verdict means the same thing for every group: of the people it flags as high risk, the same share really are, whatever their group., calibration). Individual fairnessindividual fairness, n.: The idea that a fair system should treat similar people similarly: two applicants who are alike in every way that matters should get similar outcomes. requires similar people to be treated similarly, and causal criteria such as counterfactual fairnesscounterfactual fairness, n.: A fairness test that asks: had this person been of another race or sex, with everything that would have followed from that, would the system have decided the same way? ask whether a decision would change had the person's group been different. Interventions act before training (reweighting or repairing data), during it (fairness constraints or regularizers) or after it (group-specific thresholds). Because the main group criteria cannot all hold at once (fairness impossibility theoremfairness impossibility theorem, n.: A mathematical result showing that when groups differ in how often an outcome really occurs, no imperfect prediction system can be fair by every common measure at once.), choosing among them is a normative and legal decision, not a purely technical one.
We report three fairness metrics because the regulator and the advocacy groups each care about a different one.
Origin
Grew out of discrimination-aware data mining (Dino Pedreschi, Salvatore Ruggieri and Franco Turini, 2008) and Cynthia Dwork and colleagues' 'Fairness Through Awareness' (2012). The FAT/ML workshop (from 2014) and the conference founded in 2018 as FAT* (now ACM FAccT) made it a research field; the COMPASCOMPAS, prop. n.: Commercial software used in parts of the US justice system to rate how likely someone is to commit another crime; a 2016 news investigation asking whether it was unfair to Black defendants made it famous. dispute of 2016 gave it public urgency.
Usage note
Formal metrics capture only part of fairness. Critics note that they take the decision task, the group categories and the outcome labels as given, and can ignore historical injustice, power and whether a system should be built at all. More than 20 formal definitions have been catalogued.
Related
- algorithmic biasalgorithmic bias, n.: Unfairness in software that judges or serves people: it consistently produces worse results for some groups, such as women, older people or ethnic minorities, often because it learned from skewed or incomplete records.
- group fairnessgroup fairness, n.: Measuring fairness by comparing how a system treats whole groups, such as women and men, on average: their approval rates, their error rates or what their scores mean.
- individual fairnessindividual fairness, n.: The idea that a fair system should treat similar people similarly: two applicants who are alike in every way that matters should get similar outcomes.
- counterfactual fairnesscounterfactual fairness, n.: A fairness test that asks: had this person been of another race or sex, with everything that would have followed from that, would the system have decided the same way?
- fairness impossibility theoremfairness impossibility theorem, n.: A mathematical result showing that when groups differ in how often an outcome really occurs, no imperfect prediction system can be fair by every common measure at once.
- disparate impactdisparate impact, n.: When a rule or system that treats everyone alike on paper still lands much harder on one group, such as women or older workers; the law can forbid this even if no one intended it.
- protected attributeprotected attribute, n.: A personal trait, such as race, sex, religion, age or disability, that the law says must not be used to treat people unfairly.
Sources
- Barocas, S., Hardt, M. & Narayanan, A. (2023). Fairness and Machine Learning: Limitations and Opportunities. MIT Press.
- Pedreschi, D., Ruggieri, S. & Turini, F. (2008). Discrimination-aware data mining. Proc. KDD 2008, 560–568.
Mentioned in
AI ethicsAI ethics, n.: The study of what is right and wrong in how AI is built and used: who it may harm, who is responsible when it fails, and which values it should respect., algorithmic biasalgorithmic bias, n.: Unfairness in software that judges or serves people: it consistently produces worse results for some groups, such as women, older people or ethnic minorities, often because it learned from skewed or incomplete records., biasbias, n.: A word with several AI meanings: an adjustable number added inside a model; a systematic error in estimates; or unfair treatment of some groups of people by an AI system., calibrationcalibration, n.: How well a model's stated confidence matches reality: when a well-calibrated system says it is 80% sure, it turns out to be right about 80% of the time., disparate impactdisparate impact, n.: When a rule or system that treats everyone alike on paper still lands much harder on one group, such as women or older workers; the law can forbid this even if no one intended it., fairness impossibility theoremfairness impossibility theorem, n.: A mathematical result showing that when groups differ in how often an outcome really occurs, no imperfect prediction system can be fair by every common measure at once., Joy BuolamwiniJoy Buolamwini, prop. n.: A Canadian-American computer scientist and activist whose research showed that face-analysis software works far worse on darker-skinned women, and who founded the Algorithmic Justice League., protected attributeprotected attribute, n.: A personal trait, such as race, sex, religion, age or disability, that the law says must not be used to treat people unfairly., UNESCO Recommendation on the Ethics of AIUNESCO Recommendation on the Ethics of AI, prop. n.: A worldwide set of ethical principles for AI that all of UNESCO's member countries agreed in 2021, covering human rights, fairness, privacy, the environment, gender equality and education., algorithmic recoursealgorithmic recourse, n.: Giving a person who received an unfavorable automated decision a realistic set of actions they could take to get a better result next time, such as paying down a particular debt., Arvind NarayananArvind Narayanan, prop. n.: An Indian-American computer scientist at Princeton who studies AI's real-world effects and argues that AI is a 'normal technology' whose impact will unfold gradually., counterfactual fairnesscounterfactual fairness, n.: A fairness test that asks: had this person been of another race or sex, with everything that would have followed from that, would the system have decided the same way?, Cynthia DworkCynthia Dwork, prop. n.: An American computer scientist who co-invented differential privacy, the leading mathematical definition of privacy in data analysis, and helped found the study of fairness in algorithms., FAccTFAccT, prop. n.: A yearly academic conference on the fairness, accountability and transparency of computer systems, where much influential research on AI bias and ethics has been published., group fairnessgroup fairness, n.: Measuring fairness by comparing how a system treats whole groups, such as women and men, on average: their approval rates, their error rates or what their scores mean., individual fairnessindividual fairness, n.: The idea that a fair system should treat similar people similarly: two applicants who are alike in every way that matters should get similar outcomes., Simpson's paradoxSimpson's paradox, n.: A statistics trap in which combining groups reverses what each group shows: a treatment can do better than another for men and for women separately, yet worse when everyone is counted together., value-sensitive designvalue-sensitive design, n.: A way of designing technology that deliberately identifies the human values at stake, such as privacy, fairness or autonomy, and builds them in from the start instead of patching problems later.