SmoothGrad
In plain wordsA trick for making explanation heat maps less speckled: compute the map for many noisy copies of the input and average them, so random flicker cancels out.
A technique that reduces visual noise in gradient-based saliency mapssaliency map, n.: A heat map laid over an image or text that highlights which parts most influenced an AI model's output — or, in vision science, which parts of a scene would most catch a viewer's eye. by averaging the maps computed for many copies of the input, each perturbed with Gaussian noise. Gradients of deep networks fluctuate sharply at small scales, so a single map looks speckled even when the decision depends on coherent regions; averaging over a few dozen noisy samples, with noise of roughly 10–20% of the input range, smooths these fluctuations. SmoothGrad can wrap any gradient-based method, including integrated gradientsintegrated gradients, n.: A way to score how much each input feature contributed to a neural network's output by gradually morphing a blank 'baseline' input into the real one and adding up the network's sensitivity along the way., and adding noise during training further sharpens the maps. It improves the visual coherence of maps but does not by itself establish their faithfulnessfaithfulness, n.: How accurately an explanation reflects the real reasons a model produced its output — or, for generated text, how closely it sticks to the source it was given..
Raw gradients looked like static, but with SmoothGrad the outline of the bird emerged.
Origin
Introduced by Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas and Martin Wattenberg of Google in 'SmoothGrad: removing noise by adding noise' (arXiv, June 2017).
Related
- saliency mapsaliency map, n.: A heat map laid over an image or text that highlights which parts most influenced an AI model's output — or, in vision science, which parts of a scene would most catch a viewer's eye.
- integrated gradientsintegrated gradients, n.: A way to score how much each input feature contributed to a neural network's output by gradually morphing a blank 'baseline' input into the real one and adding up the network's sensitivity along the way.
- feature attributionfeature attribution, n.: Giving each input to an AI model — each pixel, word or data column — a score for how much it pushed a particular output up or down.
- Grad-CAMGrad-CAM, abbr.: A technique that shows, as a coarse heat map, which regions of an image a vision network relied on when choosing a particular label.
- faithfulnessfaithfulness, n.: How accurately an explanation reflects the real reasons a model produced its output — or, for generated text, how closely it sticks to the source it was given.
Sources
- Smilkov, D., Thorat, N., Kim, B., Viégas, F. & Wattenberg, M. (2017). SmoothGrad: removing noise by adding noise. arXiv:1706.03825. arxiv.org/abs/1706.03825
Mentioned in
feature attributionfeature attribution, n.: Giving each input to an AI model — each pixel, word or data column — a score for how much it pushed a particular output up or down., integrated gradientsintegrated gradients, n.: A way to score how much each input feature contributed to a neural network's output by gradually morphing a blank 'baseline' input into the real one and adding up the network's sensitivity along the way., saliency mapsaliency map, n.: A heat map laid over an image or text that highlights which parts most influenced an AI model's output — or, in vision science, which parts of a scene would most catch a viewer's eye.