feature importance
In plain wordsA score for how much each input — age, income, location — matters to a model's predictions. It helps people understand, debug and simplify models, but can mislead if read as cause and effect.
A score assigned to each input feature quantifying its contribution to a model's predictions or performance. Common measures are impurity-based importance in tree ensembles (the total reduction in Gini impurityGini impurity, n.: A score for how mixed a group is: zero when every item belongs to the same category, higher when categories are jumbled together. Decision trees choose splits that make groups purer. or squared error from splits on the feature, also called mean decrease in impurity or 'gain'); permutation importance (the drop in a validation metric when the feature's values are shuffled, breaking its link to the target); the size of standardized coefficients in linear models; and mean absolute SHAPSHAP, abbr.: A popular method that explains a single prediction by fairly dividing it among the inputs, using a rule from game theory for splitting a team's winnings among its players. values, which aggregate per-prediction attributionsfeature attribution, n.: Giving each input to an AI model — each pixel, word or data column — a score for how much it pushed a particular output up or down.. Impurity-based scores are biased toward continuous and high-cardinality features and are computed on training data; permutation and SHAP importances are more reliable but split credit arbitrarily among correlated features. Importance describes the model, not the world: a feature can matter to the model without causing the outcome, and different measures can rank features very differently.
Permutation importance showed the model leaned on the patient ID — a sign of leakage.
Origin
Leo Breiman and colleagues' CART (1984) defined variable importance from surrogate splits, and Breiman's 'Random Forests' (2001) introduced permutation importance using out-of-bag data. Scott Lundberg and Su-In Lee's SHAP (2017) unified many attribution methods, and Aaron Fisher, Cynthia Rudin and Francesca Dominici generalized permutation importance as 'model reliance' (2019).
Usage note
For deep networks, gradient-based saliency mapssaliency map, n.: A heat map laid over an image or text that highlights which parts most influenced an AI model's output — or, in vision science, which parts of a scene would most catch a viewer's eye. and integrated gradientsintegrated gradients, n.: A way to score how much each input feature contributed to a neural network's output by gradually morphing a blank 'baseline' input into the real one and adding up the network's sensitivity along the way. play the analogous role. Unlike feature selectionfeature selection, n.: Choosing which of the available input measurements a model should use and dropping the rest, so the model becomes simpler, faster and often more accurate., importance scores do not by themselves remove features.
Related
- SHAPSHAP, abbr.: A popular method that explains a single prediction by fairly dividing it among the inputs, using a rule from game theory for splitting a team's winnings among its players.
- feature attributionfeature attribution, n.: Giving each input to an AI model — each pixel, word or data column — a score for how much it pushed a particular output up or down.
- random forestrandom forest, n.: A prediction method that builds hundreds of decision trees, each from a random sample of the data and features, and lets them vote. The crowd is far more reliable than any single tree.
- Gini impurityGini impurity, n.: A score for how mixed a group is: zero when every item belongs to the same category, higher when categories are jumbled together. Decision trees choose splits that make groups purer.
- feature selectionfeature selection, n.: Choosing which of the available input measurements a model should use and dropping the rest, so the model becomes simpler, faster and often more accurate.
- explainable AIexplainable AI, n.: The field and set of methods that let people understand why an AI system produced a particular output — for example, which factors led it to reject a loan application.
- partial dependence plotpartial dependence plot, n.: A chart showing how a model's average prediction changes as one input is varied while everything else stays as it is in the data — for example, predicted house price against floor area.
Sources
- Breiman, L. (2001). Random Forests. Machine Learning, 45(1), 5–32. stat.berkeley.edu/~breiman/randomforest2001.pdf
- Lundberg, S. M. & Lee, S.-I. (2017). A Unified Approach to Interpreting Model Predictions. Advances in Neural Information Processing Systems 30.
Mentioned in
featurefeature, n.: One piece of information about each example that a model uses, like a house's size or a pixel's color, or a pattern a model has learned to detect inside itself., feature attributionfeature attribution, n.: Giving each input to an AI model — each pixel, word or data column — a score for how much it pushed a particular output up or down., feature selectionfeature selection, n.: Choosing which of the available input measurements a model should use and dropping the rest, so the model becomes simpler, faster and often more accurate., Gini impurityGini impurity, n.: A score for how mixed a group is: zero when every item belongs to the same category, higher when categories are jumbled together. Decision trees choose splits that make groups purer., global explanationglobal explanation, n.: An explanation of how an AI model behaves overall — which factors matter most across all its decisions, and in what way — rather than why it made one particular decision., partial dependence plotpartial dependence plot, n.: A chart showing how a model's average prediction changes as one input is varied while everything else stays as it is in the data — for example, predicted house price against floor area., random forestrandom forest, n.: A prediction method that builds hundreds of decision trees, each from a random sample of the data and features, and lets them vote. The crowd is far more reliable than any single tree., accumulated local effectsaccumulated local effects, n.: A chart of how a model's predictions change with one input, built from small local comparisons so it stays reliable when inputs are related to each other, unlike simpler average-based charts., model-agnosticmodel-agnostic, adj.: Describes a method that works with any kind of AI model, because it only needs to feed the model inputs and observe its outputs, never to look inside., Rashomon effectRashomon effect, n.: The common situation in which many quite different models fit the same data almost equally well — like witnesses telling different but equally believable stories about the same event.