occlusion sensitivity
In plain wordsFinding out which parts of an image matter to a model by covering them up one patch at a time and watching how much the model's answer changes.
A perturbation-based feature attributionfeature attribution, n.: Giving each input to an AI model — each pixel, word or data column — a score for how much it pushed a particular output up or down. method that slides an occluding patch — gray, black, blurred or noise — across the input and records how the model's score for a target class falls at each position, giving a heat map in which large drops mark important regions. The idea extends to other data by masking words, time steps or groups of tabular features. It needs only forward passes, so it is model-agnosticmodel-agnostic, adj.: Describes a method that works with any kind of AI model, because it only needs to feed the model inputs and observe its outputs, never to look inside. and easy to explain, but it is expensive (one forward pass per patch position), sensitive to patch size and fill value, and can mislead because occluded images lie outside the training distribution, so the model may react to the artificial patch itself. Randomized-mask variants average over many random occlusions, and SHAPSHAP, abbr.: A popular method that explains a single prediction by fairly dividing it among the inputs, using a rule from game theory for splitting a team's winnings among its players. and LIMELIME, abbr.: A method that explains one prediction of any AI model by testing slightly altered versions of the input and fitting a very simple model that copies the AI's behavior near that example. can be seen as principled ways of aggregating many occlusions.
Occlusion sensitivity showed the score for 'dog' collapsing only when the patch covered the dog's face.
Origin
Popularized for deep networks by Matthew D. Zeiler and Rob Fergus in 'Visualizing and Understanding Convolutional Networks' (arXiv, November 2013; ECCV 2014), who showed that covering a dog's face, rather than its surroundings, made a classifier lose confidence in 'dog'.
Related
- feature attributionfeature attribution, n.: Giving each input to an AI model — each pixel, word or data column — a score for how much it pushed a particular output up or down.
- saliency mapsaliency map, n.: A heat map laid over an image or text that highlights which parts most influenced an AI model's output — or, in vision science, which parts of a scene would most catch a viewer's eye.
- SHAPSHAP, abbr.: A popular method that explains a single prediction by fairly dividing it among the inputs, using a rule from game theory for splitting a team's winnings among its players.
- LIMELIME, abbr.: A method that explains one prediction of any AI model by testing slightly altered versions of the input and fitting a very simple model that copies the AI's behavior near that example.
- ablationablation, n.: Removing or switching off a part of an AI system to see what changes — whether an ingredient of an experiment's design or a single unit inside a trained network.
- model-agnosticmodel-agnostic, adj.: Describes a method that works with any kind of AI model, because it only needs to feed the model inputs and observe its outputs, never to look inside.
Sources
- Zeiler, M. D. & Fergus, R. (2014). Visualizing and Understanding Convolutional Networks. Proceedings of ECCV 2014.