proxy discrimination
In plain wordsDiscrimination that happens when a system cannot see a trait like race directly but relies on something closely linked to it, such as a postcode, and so treats groups unequally anyway.
Discrimination in which a decision rule disadvantages a group defined by a protected attributeprotected attribute, n.: A personal trait, such as race, sex, religion, age or disability, that the law says must not be used to treat people unfairly. by relying on other variables correlated with it, such as postcode for race, first name for sex or ethnicity, or gaps in employment for disability or pregnancy. Machine-learning models are especially prone to it because they search large feature sets for whatever predicts the target and can reconstruct a protected attribute from many weak correlates (redundant encoding), even when it is excluded. Legal scholars focus on proxies whose predictive value comes from their correlation with the protected attribute itself, as distinct from features with independent, legitimate predictive value that merely correlate with it. Detection methods test whether the attribute can be predicted from a model's inputs or internal representations; legal challenges usually rely on disparate impactdisparate impact, n.: When a rule or system that treats everyone alike on paper still lands much harder on one group, such as women or older workers; the law can forbid this even if no one intended it. doctrine.
Dropping race did nothing: the model rebuilt it from zip code and surname, a textbook case of proxy discrimination.
Origin
The historical model is redlining, the 1930s US practice of denying mortgages using neighborhood maps that tracked race. For AI, the concept was developed in law by Anya Prince and Daniel Schwarcz ('Proxy Discrimination in the Age of Artificial Intelligence and Big Data', Iowa Law Review, 2020) and formalized in computer science by Anupam Datta and colleagues from 2017.
Related
- fairness through unawarenessfairness through unawareness, n.: The simple idea of making a system fair by hiding sensitive traits like race or sex from it; it usually fails, because other information gives those traits away.
- disparate impactdisparate impact, n.: When a rule or system that treats everyone alike on paper still lands much harder on one group, such as women or older workers; the law can forbid this even if no one intended it.
- algorithmic biasalgorithmic bias, n.: Unfairness in software that judges or serves people: it consistently produces worse results for some groups, such as women, older people or ethnic minorities, often because it learned from skewed or incomplete records.
- protected attributeprotected attribute, n.: A personal trait, such as race, sex, religion, age or disability, that the law says must not be used to treat people unfairly.
- group fairnessgroup fairness, n.: Measuring fairness by comparing how a system treats whole groups, such as women and men, on average: their approval rates, their error rates or what their scores mean.
- fairness through unawarenessfairness through unawareness, n.: The simple idea of making a system fair by hiding sensitive traits like race or sex from it; it usually fails, because other information gives those traits away.
Sources
- Prince, A. E. R. & Schwarcz, D. (2020). Proxy Discrimination in the Age of Artificial Intelligence and Big Data. Iowa Law Review 105, 1257–1318.
Mentioned in
algorithmic biasalgorithmic bias, n.: Unfairness in software that judges or serves people: it consistently produces worse results for some groups, such as women, older people or ethnic minorities, often because it learned from skewed or incomplete records., disparate impactdisparate impact, n.: When a rule or system that treats everyone alike on paper still lands much harder on one group, such as women or older workers; the law can forbid this even if no one intended it., protected attributeprotected attribute, n.: A personal trait, such as race, sex, religion, age or disability, that the law says must not be used to treat people unfairly., fairness through unawarenessfairness through unawareness, n.: The simple idea of making a system fair by hiding sensitive traits like race or sex from it; it usually fails, because other information gives those traits away.