Skip to content
Last Dictionary
Plane
Interpretability & SteeringDeep LearningMathematics & Statistics

integrated gradients

nounAlso: IG, path-integrated gradients, expected gradients

In plain wordsA way to score how much each input feature contributed to a neural network's output by gradually morphing a blank 'baseline' input into the real one and adding up the network's sensitivity along the way.

  1. A feature attributionfeature attribution, n.: Giving each input to an AI model — each pixel, word or data column — a score for how much it pushed a particular output up or down. method for differentiable models that attributes the difference between the output at an input x and at a baseline x′ (such as a black image or an all-zero embedding) to individual features, by integrating the gradientgradient, n.: A list of slopes showing which way, and how steeply, a quantity rises as each of its inputs changes. Training nudges an AI's settings in the opposite direction, downhill. of the output along the straight-line path from x′ to x and multiplying by each feature's displacement. The integral is approximated by a Riemann sum, typically over 20 to 300 steps. Integrated gradients satisfies completeness — attributions sum exactly to F(x) − F(x′) — as well as sensitivity (a feature whose change alone alters the output receives nonzero attribution) and implementation invariance (functionally identical networks receive identical attributions), axioms that plain gradients and several propagation methods violate. It corresponds to the Aumann–Shapley cost-sharing method of economics. Results depend heavily on the baseline; averaging over many baselines drawn from data gives the expected-gradients variant.

    With a black-image baseline, integrated gradients gave almost no credit to the dark pixels of the tumor.

IGi(x)=(xi−xi′)∫01∂F(x′+α(x−x′))∂xi dα\mathrm{IG}_i(x) = (x_i - x'_i)\int_{0}^{1} \frac{\partial F\big(x' + \alpha (x - x')\big)}{\partial x_i}\, d\alpha

Origin

Introduced by Mukund Sundararajan, Ankur Taly and Qiqi Yan of Google in 'Axiomatic Attribution for Deep Networks' (arXiv, March 2017; ICML 2017).

Usage note

No baseline is neutral for every task — a black baseline cannot attribute importance to black pixels. Like other attribution methods, integrated gradients measures this model's sensitivity, not causal importance in the world.

Related

See also
  • feature attributionfeature attribution, n.: Giving each input to an AI model — each pixel, word or data column — a score for how much it pushed a particular output up or down.
  • saliency mapsaliency map, n.: A heat map laid over an image or text that highlights which parts most influenced an AI model's output — or, in vision science, which parts of a scene would most catch a viewer's eye.
  • SHAPSHAP, abbr.: A popular method that explains a single prediction by fairly dividing it among the inputs, using a rule from game theory for splitting a team's winnings among its players.
  • Shapley valueShapley value, n.: A rule from game theory for splitting a shared reward fairly among team members, by averaging how much each member adds when joining the team in every possible order.
  • SmoothGradSmoothGrad, n.: A trick for making explanation heat maps less speckled: compute the map for many noisy copies of the input and average them, so random flicker cancels out.
  • DeepLIFTDeepLIFT, abbr.: A method that explains a neural network's output by comparing each internal signal with its value on a neutral reference input, then passing the differences back to the inputs as credit.

Sources

  1. Sundararajan, M., Taly, A. & Yan, Q. (2017). Axiomatic Attribution for Deep Networks. Proceedings of ICML 2017.

Mentioned in

explainable AIexplainable AI, n.: The field and set of methods that let people understand why an AI system produced a particular output — for example, which factors led it to reject a loan application., baselinebaseline, n.: A simple reference method that a new model must beat to show it is worth anything, such as always guessing the most common answer or reusing yesterday's value., feature attributionfeature attribution, n.: Giving each input to an AI model — each pixel, word or data column — a score for how much it pushed a particular output up or down., feature importancefeature importance, n.: A score for how much each input — age, income, location — matters to a model's predictions. It helps people understand, debug and simplify models, but can mislead if read as cause and effect., Grad-CAMGrad-CAM, abbr.: A technique that shows, as a coarse heat map, which regions of an image a vision network relied on when choosing a particular label., gradientgradient, n.: A list of slopes showing which way, and how steeply, a quantity rises as each of its inputs changes. Training nudges an AI's settings in the opposite direction, downhill., local explanationlocal explanation, n.: An explanation of why an AI model gave one particular answer for one particular case — this patient, this loan — rather than how it behaves in general., post hoc explanationpost hoc explanation, n.: An explanation produced after a model has been built, by a separate tool that examines the finished model, rather than one that comes from a model designed to be understandable., saliency mapsaliency map, n.: A heat map laid over an image or text that highlights which parts most influenced an AI model's output — or, in vision science, which parts of a scene would most catch a viewer's eye., Shapley valueShapley value, n.: A rule from game theory for splitting a shared reward fairly among team members, by averaging how much each member adds when joining the team in every possible order., attribution patchingattribution patching, n.: A fast shortcut that estimates the results of thousands of patching experiments at once, using the slope of the model's output instead of rerunning the model for every part., DeepLIFTDeepLIFT, abbr.: A method that explains a neural network's output by comparing each internal signal with its value on a neutral reference input, then passing the differences back to the inputs as credit., knowledge neuronknowledge neuron, n.: An internal unit of a language model claimed to be responsible for storing a particular fact, such that turning it up or down makes the model more or less likely to state that fact., layer-wise relevance propagationlayer-wise relevance propagation, n.: A method that traces a neural network's decision backward, layer by layer, handing each part of the input its share of the credit until every pixel or word has a relevance score., model-agnosticmodel-agnostic, adj.: Describes a method that works with any kind of AI model, because it only needs to feed the model inputs and observe its outputs, never to look inside., SmoothGradSmoothGrad, n.: A trick for making explanation heat maps less speckled: compute the map for many noisy copies of the input and average them, so random flicker cancels out.