counterfactual explanation
In plain wordsAn explanation that tells someone the smallest change that would have produced a different outcome — for example, that the application would have been accepted if it had listed two more years of work experience.
An explanation of an individual decision in the form of a minimal change to the input that would have changed the model's output to a specified alternative: if X had been different, the outcome would have been Y. Generating one is an optimization problem: find a point x′ close to the original input x — under a distance that favors changing few features — for which the model returns the desired output, usually with constraints that x′ be plausible under the data distribution and alter only mutable features. Desiderata include proximity, sparsity, plausibility, actionability, diversity (offering several alternatives) and causal consistency (respecting how features affect one another). Counterfactuals need no access to model internals and can be understood without technical training, which made them attractive under data-protection law; but they are not unique, may not survive retraining, and say little about the model as a whole.
The counterfactual explanation said two fewer open credit lines would have flipped the decision.
Origin
Popularized in machine learning by Sandra Wachter, Brent Mittelstadt and Chris Russell in 'Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR' (arXiv, 2017; Harvard Journal of Law & Technology, 2018), drawing on philosophical accounts of counterfactual causation such as David Lewis's (1973). David Martens and Foster Provost (2014) had already explained document classifications by the smallest set of words whose removal changes the prediction.
Usage note
Distinguish counterfactual fairnesscounterfactual fairness, n.: A fairness test that asks: had this person been of another race or sex, with everything that would have followed from that, would the system have decided the same way?, a fairness criterion, and the general notion of a counterfactualcounterfactual, n.: A statement about what would have happened if something had been different — 'had I left earlier, I would have caught the train'. AI uses counterfactuals to explain decisions and to measure causes.. A counterfactual explanation states what would flip this model's decision, not what the person ought to do; turning it into feasible advice is the aim of algorithmic recoursealgorithmic recourse, n.: Giving a person who received an unfavorable automated decision a realistic set of actions they could take to get a better result next time, such as paying down a particular debt..
Related
- algorithmic recoursealgorithmic recourse, n.: Giving a person who received an unfavorable automated decision a realistic set of actions they could take to get a better result next time, such as paying down a particular debt.
- local explanationlocal explanation, n.: An explanation of why an AI model gave one particular answer for one particular case — this patient, this loan — rather than how it behaves in general.
- right to explanationright to explanation, n.: The idea that a person affected by a decision made or shaped by an algorithm — a refused loan, a rejected job application — is entitled to a meaningful account of why.
- counterfactualcounterfactual, n.: A statement about what would have happened if something had been different — 'had I left earlier, I would have caught the train'. AI uses counterfactuals to explain decisions and to measure causes.
- model-agnosticmodel-agnostic, adj.: Describes a method that works with any kind of AI model, because it only needs to feed the model inputs and observe its outputs, never to look inside.
- feature attributionfeature attribution, n.: Giving each input to an AI model — each pixel, word or data column — a score for how much it pushed a particular output up or down.
- counterfactual fairnesscounterfactual fairness, n.: A fairness test that asks: had this person been of another race or sex, with everything that would have followed from that, would the system have decided the same way?
Sources
- Wachter, S., Mittelstadt, B. & Russell, C. (2018). Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR. Harvard Journal of Law & Technology 31(2), 841–887.
- Mothilal, R. K., Sharma, A. & Tan, C. (2020). Explaining Machine Learning Classifiers through Diverse Counterfactual Explanations. Proceedings of ACM FAT* 2020.
Mentioned in
explainable AIexplainable AI, n.: The field and set of methods that let people understand why an AI system produced a particular output — for example, which factors led it to reject a loan application., counterfactualcounterfactual, n.: A statement about what would have happened if something had been different — 'had I left earlier, I would have caught the train'. AI uses counterfactuals to explain decisions and to measure causes., local explanationlocal explanation, n.: An explanation of why an AI model gave one particular answer for one particular case — this patient, this loan — rather than how it behaves in general., post hoc explanationpost hoc explanation, n.: An explanation produced after a model has been built, by a separate tool that examines the finished model, rather than one that comes from a model designed to be understandable., right to explanationright to explanation, n.: The idea that a person affected by a decision made or shaped by an algorithm — a refused loan, a rejected job application — is entitled to a meaningful account of why., algorithmic recoursealgorithmic recourse, n.: Giving a person who received an unfavorable automated decision a realistic set of actions they could take to get a better result next time, such as paying down a particular debt., AnchorsAnchors, prop. n.: A method that explains a prediction with a simple if-then rule that locks in the result — for example, if an email contains both 'invoice' and 'urgent', the filter almost always flags it., model-agnosticmodel-agnostic, adj.: Describes a method that works with any kind of AI model, because it only needs to feed the model inputs and observe its outputs, never to look inside.