post hoc explanation
In plain wordsAn explanation produced after a model has been built, by a separate tool that examines the finished model, rather than one that comes from a model designed to be understandable.
An explanation of a trained model's behavior generated by a method applied after training, without constraining how the model was built. Post hoc methods may treat the model as a black boxblack box, n.: A system whose inner workings are hidden or too complex to understand: we can see what goes in and what comes out, but not why it gave a particular answer. that can only be queried (LIMELIME, abbr.: A method that explains one prediction of any AI model by testing slightly altered versions of the input and fitting a very simple model that copies the AI's behavior near that example., SHAPSHAP, abbr.: A popular method that explains a single prediction by fairly dividing it among the inputs, using a rule from game theory for splitting a team's winnings among its players., counterfactualscounterfactual explanation, n.: An explanation that tells someone the smallest change that would have produced a different outcome — for example, that the application would have been accepted if it had listed two more years of work experience., partial dependencepartial dependence plot, n.: A chart showing how a model's average prediction changes as one input is varied while everything else stays as it is in the data — for example, predicted house price against floor area.) or exploit its internals (gradient saliencysaliency map, n.: A heat map laid over an image or text that highlights which parts most influenced an AI model's output — or, in vision science, which parts of a scene would most catch a viewer's eye., integrated gradientsintegrated gradients, n.: A way to score how much each input feature contributed to a neural network's output by gradually morphing a blank 'baseline' input into the real one and adding up the network's sensitivity along the way., Grad-CAMGrad-CAM, abbr.: A technique that shows, as a coarse heat map, which regions of an image a vision network relied on when choosing a particular label., concept activation vectorstesting with concept activation vectors, phr.: A method for asking whether a human idea, such as 'wheels', matters to an image model's decision, such as 'car', by finding the direction for that idea inside the model and measuring its effect., probingprobing, n.: Testing what information is hidden inside an AI model by training a small, separate predictor on the model's internal activity: if the predictor can recover something, the model has stored it somewhere.). They can be applied to existing models too complex to understand directly, but they approximate rather than reveal the model's computation: different methods often disagree about the same prediction, explanations can shift under tiny input changes, and a model can be deliberately built to hide discriminatory behavior from perturbation-based explainers. Their reliability is therefore checked with faithfulnessfaithfulness, n.: How accurately an explanation reflects the real reasons a model produced its output — or, for generated text, how closely it sticks to the source it was given. tests, such as deleting the features an explanation ranks highest and measuring how much the output changes.
The model was a gradient-boosted ensemble, so we relied on post hoc explanations from SHAP.
Origin
Zachary C. Lipton's 'The Mythos of Model Interpretability' (2016) contrasted post hoc interpretability with transparency; the post hoc versus intrinsic (or ante hoc) distinction became standard in surveys by 2018–2019 and was sharpened by Cynthia Rudin's 2019 critique of explaining black boxes in high-stakes settings.
Usage note
Latin post hoc, 'after this'. Rudin argued that post hoc explanations can mislead and that interpretable models often perform as well on structured data; defenders reply that many high-performing models, including all large language models, admit no simpler substitute.
Related
- interpretable modelinterpretable model, n.: A model simple or structured enough that a person can follow exactly how it turns inputs into a decision — like a short checklist or a points-based score — with no separate explanation tool needed.
- explainable AIexplainable AI, n.: The field and set of methods that let people understand why an AI system produced a particular output — for example, which factors led it to reject a loan application.
- feature attributionfeature attribution, n.: Giving each input to an AI model — each pixel, word or data column — a score for how much it pushed a particular output up or down.
- surrogate modelsurrogate model, n.: A simpler stand-in model trained to imitate a more complex one — to explain it, or to approximate a slow or costly calculation cheaply.
- faithfulnessfaithfulness, n.: How accurately an explanation reflects the real reasons a model produced its output — or, for generated text, how closely it sticks to the source it was given.
- local explanationlocal explanation, n.: An explanation of why an AI model gave one particular answer for one particular case — this patient, this loan — rather than how it behaves in general.
- interpretable modelinterpretable model, n.: A model simple or structured enough that a person can follow exactly how it turns inputs into a decision — like a short checklist or a points-based score — with no separate explanation tool needed.
Sources
- Lipton, Z. C. (2018). The Mythos of Model Interpretability. ACM Queue 16(3). (Workshop version 2016.)
- Rudin, C. (2019). Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence 1, 206–215.
Mentioned in
explainabilityexplainability, n.: How well the reasons behind an AI system's outputs can be put into terms that people can understand, check and, if needed, challenge., explainable AIexplainable AI, n.: The field and set of methods that let people understand why an AI system produced a particular output — for example, which factors led it to reject a loan application., interpretabilityinterpretability, n.: The effort to understand how an AI system arrives at its outputs — what information it uses and how it combines it — instead of treating it as a sealed black box., faithfulnessfaithfulness, n.: How accurately an explanation reflects the real reasons a model produced its output — or, for generated text, how closely it sticks to the source it was given., interpretable modelinterpretable model, n.: A model simple or structured enough that a person can follow exactly how it turns inputs into a decision — like a short checklist or a points-based score — with no separate explanation tool needed., LIMELIME, abbr.: A method that explains one prediction of any AI model by testing slightly altered versions of the input and fitting a very simple model that copies the AI's behavior near that example., chain-of-thought faithfulnesschain-of-thought faithfulness, n.: Whether the reasoning an AI writes out really reflects how it reached its answer, rather than being an after-the-fact story that leaves out what actually drove the decision., concept bottleneck modelconcept bottleneck model, n.: A model made to first predict human-understandable concepts — such as 'bone spur present' on an X-ray — and then decide only from those concepts, so people can inspect and correct them., model-agnosticmodel-agnostic, adj.: Describes a method that works with any kind of AI model, because it only needs to feed the model inputs and observe its outputs, never to look inside., prototypical part networkprototypical part network, n.: An image classifier that explains each decision by pointing to parts of the picture that resemble typical examples it learned — this part looks like that part of a known bird.