Skip to content
Last Dictionary
Plane
Interpretability & SteeringEvaluation & BenchmarksSafety & Alignment

interpretability illusion

nounAlso: illusion of interpretability, interpretability illusions

In plain wordsA convincing-looking explanation of what is going on inside an AI model that turns out to be wrong or misleading, often because the evidence examined was too narrow.

  1. An apparently compelling interpretation of a model's internals that fails under broader or more rigorous testing. Classic forms include a neuron or direction that seems to encode one clean concept when inspected on one dataset but encodes different concepts on others (Tolga Bolukbasi and colleagues, 2021); subspace activation patchingactivation patching, n.: An experiment that copies part of a model's internal state from its run on one input into its run on another, to see which internal parts actually cause a change in the answer. that changes outputs by activating a dormant pathway the model does not normally use, rather than the variable the subspace supposedly represents (Aleksandar Makelov, Georg Lange and Neel Nanda, 2023); and simplified proxy models or circuits that match the original on test data but generalize differently. Related pitfalls include labeling features only from their top activations, mistaking decodability by a probeprobing, n.: Testing what information is hidden inside an AI model by training a small, separate predictor on the model's internal activity: if the predictor can recover something, the model has stored it somewhere. for use by the model, and visualizations that look meaningful even for randomly initialized networks. Defenses include testing on varied data, causal interventions with controls and quantitative measures of faithfulness.

    The 'negation' direction was an interpretability illusion: on a second corpus it tracked sentence length.

Origin

Named by Tolga Bolukbasi, Adam Pearce, Ann Yuan, Andy Coenen, Emily Reif, Fernanda Viégas and Martin Wattenberg in 'An Interpretability Illusion for BERT' (arXiv, April 2021); extended to subspace patching by Makelov, Lange and Nanda (2023) and to simplified models by Dan Friedman and colleagues (2023).

Related

See also
  • activation patchingactivation patching, n.: An experiment that copies part of a model's internal state from its run on one input into its run on another, to see which internal parts actually cause a change in the answer.
  • distributed alignment searchdistributed alignment search, n.: A method that searches, by training, for a hidden direction inside a neural network that plays the role of one variable in a human-readable explanation of the network's behavior.
  • probingprobing, n.: Testing what information is hidden inside an AI model by training a small, separate predictor on the model's internal activity: if the predictor can recover something, the model has stored it somewhere.
  • feature absorptionfeature absorption, n.: A flaw in concept-finding tools where a general concept goes missing in some cases because a more specific concept swallows it — like a 'starts with S' detector staying silent on the word 'snake'.
  • causal scrubbingcausal scrubbing, n.: A strict test of an explanation of how a model works: scramble every internal signal the explanation says should not matter, and check that the model still performs as before.
  • saliency mapsaliency map, n.: A heat map laid over an image or text that highlights which parts most influenced an AI model's output — or, in vision science, which parts of a scene would most catch a viewer's eye.

Sources

  1. Bolukbasi, T., Pearce, A., Yuan, A., Coenen, A., Reif, E., Viégas, F. & Wattenberg, M. (2021). An Interpretability Illusion for BERT. arXiv preprint.
  2. Makelov, A., Lange, G. & Nanda, N. (2024). Is This the Subspace You Are Looking For? An Interpretability Illusion for Subspace Activation Patching. ICLR 2024.
  3. Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I., Hardt, M. & Kim, B. (2018). Sanity Checks for Saliency Maps. NeurIPS 2018.

Mentioned in

activation patchingactivation patching, n.: An experiment that copies part of a model's internal state from its run on one input into its run on another, to see which internal parts actually cause a change in the answer., causal abstractioncausal abstraction, n.: A mathematical way to check whether a simple, human-readable recipe really describes what a neural network does inside, by testing whether matching internal parts behave the same way when swapped., causal scrubbingcausal scrubbing, n.: A strict test of an explanation of how a model works: scramble every internal signal the explanation says should not matter, and check that the model still performs as before., distributed alignment searchdistributed alignment search, n.: A method that searches, by training, for a hidden direction inside a neural network that plays the role of one variable in a human-readable explanation of the network's behavior., feature absorptionfeature absorption, n.: A flaw in concept-finding tools where a general concept goes missing in some cases because a more specific concept swallows it — like a 'starts with S' detector staying silent on the word 'snake'., replacement modelreplacement model, n.: A more understandable copy of an AI model, built from simple labeled parts, that behaves almost like the original, so researchers can study the copy's workings instead.