COMPAS
In plain wordsCommercial software used in parts of the US justice system to rate how likely someone is to commit another crime; a 2016 news investigation asking whether it was unfair to Black defendants made it famous.
A proprietary risk-and-needs assessment instrument, developed by Northpointe (now Equivant), that scores defendants and offenders from 1 to 10 on risks such as general and violent recidivism, using questionnaire answers and criminal history; race is not an input. ProPublica's May 2016 analysis of the scores given to about 7,000 arrestees in Broward County, Florida, found unequal error rates, in breach of equalized oddsequalized odds, n.: A fairness test that asks whether a system makes mistakes at the same rates for every group: wrongly flagging innocent people, and wrongly missing real cases, equally often.: among defendants who did not reoffend within two years, 45% of Black and 23% of white defendants had been rated medium or high risk, while among those who did reoffend, white defendants were more often rated low risk. Northpointe replied that scores meant the same reoffense probability for both groups (predictive paritypredictive parity, n.: A fairness test that asks whether a system's verdict means the same thing for every group: of the people it flags as high risk, the same share really are, whatever their group.). Proofs that both conditions cannot hold together when base rates differ made the case the founding example of the fairness impossibility theoremfairness impossibility theorem, n.: A mathematical result showing that when groups differ in how often an outcome really occurs, no imperfect prediction system can be fair by every common measure at once..
Nearly every fairness course still opens with the COMPAS dataset.
Origin
Developed in 1998 by Tim Brennan and colleagues at Northpointe and adopted by many US jurisdictions. Its notoriety dates from 'Machine Bias' by Julia Angwin, Jeff Larson, Surya Mattu and Lauren Kirchner (ProPublica, 23 May 2016), published weeks before the Wisconsin Supreme Court allowed its use at sentencing, with cautions, in State v. Loomis (July 2016).
Usage note
A later study found COMPAS no more accurate than pooled predictions by untrained online volunteers, or than a simple model using only age and number of prior convictions (Dressel and Farid, 2018), widening the debate over whether such tools improve on human judgment.
Related
- fairness impossibility theoremfairness impossibility theorem, n.: A mathematical result showing that when groups differ in how often an outcome really occurs, no imperfect prediction system can be fair by every common measure at once.
- equalized oddsequalized odds, n.: A fairness test that asks whether a system makes mistakes at the same rates for every group: wrongly flagging innocent people, and wrongly missing real cases, equally often.
- predictive paritypredictive parity, n.: A fairness test that asks whether a system's verdict means the same thing for every group: of the people it flags as high risk, the same share really are, whatever their group.
- algorithmic biasalgorithmic bias, n.: Unfairness in software that judges or serves people: it consistently produces worse results for some groups, such as women, older people or ethnic minorities, often because it learned from skewed or incomplete records.
- predictive policingpredictive policing, n.: Using computer programs to forecast where crimes are likely to happen, or who is likely to be involved, so that police can decide where to patrol or whom to watch.
- algorithmic accountabilityalgorithmic accountability, n.: The idea that the people and organizations behind automated decision systems must be able to explain and justify those systems' effects, answer for the harm they cause, and fix it.
Sources
- Angwin, J., Larson, J., Mattu, S. & Kirchner, L. (2016). Machine Bias. ProPublica, 23 May 2016.
- Dressel, J. & Farid, H. (2018). The accuracy, fairness, and limits of predicting recidivism. Science Advances 4(1), eaao5580.
Mentioned in
algorithmic biasalgorithmic bias, n.: Unfairness in software that judges or serves people: it consistently produces worse results for some groups, such as women, older people or ethnic minorities, often because it learned from skewed or incomplete records., algorithmic fairnessalgorithmic fairness, n.: The effort to make automated decisions fair: to say in numbers what fair treatment means, measure whether a system achieves it, and correct the system if it does not., biasbias, n.: A word with several AI meanings: an adjustable number added inside a model; a systematic error in estimates; or unfair treatment of some groups of people by an AI system., calibrationcalibration, n.: How well a model's stated confidence matches reality: when a well-calibrated system says it is 80% sure, it turns out to be right about 80% of the time., fairness impossibility theoremfairness impossibility theorem, n.: A mathematical result showing that when groups differ in how often an outcome really occurs, no imperfect prediction system can be fair by every common measure at once., predictive AIpredictive AI, n.: AI used to forecast outcomes about people or events, such as who will repay a loan, which patient may get sicker or which machine will fail, in order to guide decisions., predictive policingpredictive policing, n.: Using computer programs to forecast where crimes are likely to happen, or who is likely to be involved, so that police can decide where to patrol or whom to watch., RobodebtRobodebt, prop. n.: An Australian government program that used an automated calculation to accuse hundreds of thousands of welfare recipients of owing money, often wrongly. An official inquiry later found it unfair and illegal., equalized oddsequalized odds, n.: A fairness test that asks whether a system makes mistakes at the same rates for every group: wrongly flagging innocent people, and wrongly missing real cases, equally often., predictive paritypredictive parity, n.: A fairness test that asks whether a system's verdict means the same thing for every group: of the people it flags as high risk, the same share really are, whatever their group.