Skip to content
Last Dictionary
Plane
Society, Ethics & EconomyMachine Learning

predictive parity

nounAlso: sufficiency (fairness criterion), calibration within groups, PPV parity, positive predictive value parity

In plain wordsA fairness test that asks whether a system's verdict means the same thing for every group: of the people it flags as high risk, the same share really are, whatever their group.

  1. A group fairnessgroup fairness, n.: Measuring fairness by comparing how a system treats whole groups, such as women and men, on average: their approval rates, their error rates or what their scores mean. criterion requiring equal positive predictive value across groups: among individuals who receive a positive prediction, the fraction whose true outcome is positive must not depend on the protected attributeprotected attribute, n.: A personal trait, such as race, sex, religion, age or disability, that the law says must not be used to treat people unfairly.. Its score-based generalization, calibration within groups (or sufficiency), requires a given risk score to correspond to the same outcome probability in every group, so that the outcome is independent of group membership given the score. It is the standard Northpointe invoked to defend COMPASCOMPAS, prop. n.: Commercial software used in parts of the US justice system to rate how likely someone is to commit another crime; a 2016 news investigation asking whether it was unfair to Black defendants made it famous.. When base rates differ, an imperfect classifier that satisfies it must have unequal false positive or false negative rates, so it conflicts with equalized oddsequalized odds, n.: A fairness test that asks whether a system makes mistakes at the same rates for every group: wrongly flagging innocent people, and wrongly missing real cases, equally often. (see fairness impossibility theoremfairness impossibility theorem, n.: A mathematical result showing that when groups differ in how often an outcome really occurs, no imperfect prediction system can be fair by every common measure at once.).

    The tool satisfies predictive parity: of those it flags as high risk, about 60% are rearrested in every group.

P(Y=1∣Y^=1,A=a)=P(Y=1∣Y^=1,A=b)P(Y=1 \mid \hat{Y}=1, A=a) = P(Y=1 \mid \hat{Y}=1, A=b)

Origin

The phrase appears in the title of Northpointe's July 2016 reply to ProPublica by William Dieterich, Christina Mendoza and Tim Brennan; Alexandra Chouldechova gave it its standard formal definition as equal positive predictive value (2016–2017). Checks of calibration across groups have a longer history in educational and psychological testing.

Related

See also
  • equalized oddsequalized odds, n.: A fairness test that asks whether a system makes mistakes at the same rates for every group: wrongly flagging innocent people, and wrongly missing real cases, equally often.
  • demographic paritydemographic parity, n.: A fairness test that asks whether a system says yes (approves the loan, shortlists the applicant) at the same rate for every group, whatever other differences exist between them.
  • fairness impossibility theoremfairness impossibility theorem, n.: A mathematical result showing that when groups differ in how often an outcome really occurs, no imperfect prediction system can be fair by every common measure at once.
  • COMPASCOMPAS, prop. n.: Commercial software used in parts of the US justice system to rate how likely someone is to commit another crime; a 2016 news investigation asking whether it was unfair to Black defendants made it famous.
  • calibrationcalibration, n.: How well a model's stated confidence matches reality: when a well-calibrated system says it is 80% sure, it turns out to be right about 80% of the time.
  • group fairnessgroup fairness, n.: Measuring fairness by comparing how a system treats whole groups, such as women and men, on average: their approval rates, their error rates or what their scores mean.
Contrast with
  • equalized oddsequalized odds, n.: A fairness test that asks whether a system makes mistakes at the same rates for every group: wrongly flagging innocent people, and wrongly missing real cases, equally often.

Sources

  1. Dieterich, W., Mendoza, C. & Brennan, T. (2016). COMPAS Risk Scales: Demonstrating Accuracy Equity and Predictive Parity. Northpointe Inc.
  2. Chouldechova, A. (2017). Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction Instruments. Big Data 5(2), 153–163.

Mentioned in

algorithmic fairnessalgorithmic fairness, n.: The effort to make automated decisions fair: to say in numbers what fair treatment means, measure whether a system achieves it, and correct the system if it does not., COMPASCOMPAS, prop. n.: Commercial software used in parts of the US justice system to rate how likely someone is to commit another crime; a 2016 news investigation asking whether it was unfair to Black defendants made it famous., fairness impossibility theoremfairness impossibility theorem, n.: A mathematical result showing that when groups differ in how often an outcome really occurs, no imperfect prediction system can be fair by every common measure at once., demographic paritydemographic parity, n.: A fairness test that asks whether a system says yes (approves the loan, shortlists the applicant) at the same rate for every group, whatever other differences exist between them., equalized oddsequalized odds, n.: A fairness test that asks whether a system makes mistakes at the same rates for every group: wrongly flagging innocent people, and wrongly missing real cases, equally often., group fairnessgroup fairness, n.: Measuring fairness by comparing how a system treats whole groups, such as women and men, on average: their approval rates, their error rates or what their scores mean.