SHAP
In plain wordsA popular method that explains a single prediction by fairly dividing it among the inputs, using a rule from game theory for splitting a team's winnings among its players.
A framework for feature attributionfeature attribution, n.: Giving each input to an AI model — each pixel, word or data column — a score for how much it pushed a particular output up or down. that explains an individual prediction as a sum of per-feature contributions equal to the features' Shapley valuesShapley value, n.: A rule from game theory for splitting a shared reward fairly among team members, by averaging how much each member adds when joining the team in every possible order., treating features as players in a cooperative game whose payout is the model's output. The explanation is additive: the base value (the average prediction) plus all features' SHAP values equals the prediction. Its authors showed that Shapley values are the unique additive attribution satisfying local accuracy, missingness and consistency, and that LIMELIME, abbr.: A method that explains one prediction of any AI model by testing slightly altered versions of the input and fitting a very simple model that copies the AI's behavior near that example., DeepLIFTDeepLIFT, abbr.: A method that explains a neural network's output by comparing each internal signal with its value on a neutral reference input, then passing the differences back to the inputs as credit. and layer-wise relevance propagationlayer-wise relevance propagation, n.: A method that traces a neural network's decision backward, layer by layer, handing each part of the input its share of the credit until every pixel or word has a relevance score. belong to the same family. Exact computation is exponential in the number of features, so practical estimators include KernelSHAP (weighted linear regression on sampled feature subsets), TreeSHAP (exact polynomial-time computation for tree ensembles) and DeepSHAP (an approximation for neural networks). The open-source shap library made it a default for explaining gradient-boostedgradient boosting, n.: A method that builds a strong predictor by adding small decision trees one at a time, each trained to correct the errors that remain after all the trees before it. models on tabular data; its summary and dependence plots also give global views.
The SHAP waterfall plot showed the short credit history pulling the applicant's score down the most.
Origin
Introduced by Scott M. Lundberg and Su-In Lee of the University of Washington in 'A Unified Approach to Interpreting Model Predictions' (NeurIPS 2017); TreeSHAP followed (arXiv, 2018; Nature Machine Intelligence, 2020). Erik Štrumbelj and Igor Kononenko had used Shapley values to explain classifier predictions from 2010.
Usage note
SHAP values depend on how 'absent' features are simulated: interventional and conditional versions can disagree when features are correlated, and neither measures causal effects in the world. A large SHAP value means a feature matters to the model, not necessarily to the outcome.
Related
- Shapley valueShapley value, n.: A rule from game theory for splitting a shared reward fairly among team members, by averaging how much each member adds when joining the team in every possible order.
- feature attributionfeature attribution, n.: Giving each input to an AI model — each pixel, word or data column — a score for how much it pushed a particular output up or down.
- LIMELIME, abbr.: A method that explains one prediction of any AI model by testing slightly altered versions of the input and fitting a very simple model that copies the AI's behavior near that example.
- gradient boostinggradient boosting, n.: A method that builds a strong predictor by adding small decision trees one at a time, each trained to correct the errors that remain after all the trees before it.
- local explanationlocal explanation, n.: An explanation of why an AI model gave one particular answer for one particular case — this patient, this loan — rather than how it behaves in general.
- partial dependence plotpartial dependence plot, n.: A chart showing how a model's average prediction changes as one input is varied while everything else stays as it is in the data — for example, predicted house price against floor area.
- LIMELIME, abbr.: A method that explains one prediction of any AI model by testing slightly altered versions of the input and fitting a very simple model that copies the AI's behavior near that example.
Sources
- Lundberg, S. M. & Lee, S.-I. (2017). A Unified Approach to Interpreting Model Predictions. NeurIPS 2017.
- Lundberg, S. M. et al. (2020). From local explanations to global understanding with explainable AI for trees. Nature Machine Intelligence 2, 56–67.
Mentioned in
explainable AIexplainable AI, n.: The field and set of methods that let people understand why an AI system produced a particular output — for example, which factors led it to reject a loan application., interpretabilityinterpretability, n.: The effort to understand how an AI system arrives at its outputs — what information it uses and how it combines it — instead of treating it as a sealed black box., baselinebaseline, n.: A simple reference method that a new model must beat to show it is worth anything, such as always guessing the most common answer or reusing yesterday's value., feature attributionfeature attribution, n.: Giving each input to an AI model — each pixel, word or data column — a score for how much it pushed a particular output up or down., feature importancefeature importance, n.: A score for how much each input — age, income, location — matters to a model's predictions. It helps people understand, debug and simplify models, but can mislead if read as cause and effect., game theorygame theory, n.: The mathematics of strategic decisions, where each player's best move depends on what the others do. It is used to study competition and cooperation among people, firms, nations and AI agents., global explanationglobal explanation, n.: An explanation of how an AI model behaves overall — which factors matter most across all its decisions, and in what way — rather than why it made one particular decision., integrated gradientsintegrated gradients, n.: A way to score how much each input feature contributed to a neural network's output by gradually morphing a blank 'baseline' input into the real one and adding up the network's sensitivity along the way., LIMELIME, abbr.: A method that explains one prediction of any AI model by testing slightly altered versions of the input and fitting a very simple model that copies the AI's behavior near that example., local explanationlocal explanation, n.: An explanation of why an AI model gave one particular answer for one particular case — this patient, this loan — rather than how it behaves in general., partial dependence plotpartial dependence plot, n.: A chart showing how a model's average prediction changes as one input is varied while everything else stays as it is in the data — for example, predicted house price against floor area., post hoc explanationpost hoc explanation, n.: An explanation produced after a model has been built, by a separate tool that examines the finished model, rather than one that comes from a model designed to be understandable., Shapley valueShapley value, n.: A rule from game theory for splitting a shared reward fairly among team members, by averaging how much each member adds when joining the team in every possible order., DeepLIFTDeepLIFT, abbr.: A method that explains a neural network's output by comparing each internal signal with its value on a neutral reference input, then passing the differences back to the inputs as credit., model-agnosticmodel-agnostic, adj.: Describes a method that works with any kind of AI model, because it only needs to feed the model inputs and observe its outputs, never to look inside., occlusion sensitivityocclusion sensitivity, n.: Finding out which parts of an image matter to a model by covering them up one patch at a time and watching how much the model's answer changes.