Gini impurity
In plain wordsA score for how mixed a group is: zero when every item belongs to the same category, higher when categories are jumbled together. Decision trees choose splits that make groups purer.
A measure of the class heterogeneity of a set of examples: the probability that two items drawn at random with replacement belong to different classes — equivalently, the expected error rate of guessing each item's class at random in proportion to the class frequencies. It is 0 for a pure node and reaches its maximum, 1 − 1/K, when K classes are equally represented. CARTCART, abbr.: A classic recipe for building decision trees that splits data two ways at every step, works for both categories and numbers, and then trims the tree back so it does not memorize noise. and most modern decision treedecision tree, n.: A model that makes a prediction by asking a series of yes-or-no questions about the input, like a flowchart, with each answer leading to the next question until it reaches a final answer. software choose the split that most reduces the size-weighted Gini impurity of the child nodes, and the summed reductions across a tree or forest give the impurity-based ('mean decrease in impurity') feature importancefeature importance, n.: A score for how much each input — age, income, location — matters to a model's predictions. It helps people understand, debug and simplify models, but can mislead if read as cause and effect.. It ranks splits much as entropy-based information gaininformation gain, n.: How much a question reduces uncertainty about the answer. A decision tree asks first the question that removes the most uncertainty, measured in bits. does but needs no logarithms.
The split on contract type cut the weighted Gini impurity from 0.48 to 0.31.
Origin
The quantity is the Gini–Simpson diversity index, associated with the Italian statistician Corrado Gini's work on variability (1912) and introduced as a concentration measure by Edward Simpson (1949). Richard Light and Barry Margolin used it to analyze categorical data (1971), and Breiman, Friedman, Olshen and Stone made it CART's default splitting criterion (1984).
Usage note
Not the same as the Gini coefficient of income inequality, also named for Gini; and in credit scoring 'Gini' usually means 2 × AUCAUC, abbr.: A single number from 0 to 1 summarizing how well a model ranks real positive cases above negative ones: 0.5 is no better than a coin flip and 1.0 is perfect. − 1, a ranking metric.
Related
- decision treedecision tree, n.: A model that makes a prediction by asking a series of yes-or-no questions about the input, like a flowchart, with each answer leading to the next question until it reaches a final answer.
- CARTCART, abbr.: A classic recipe for building decision trees that splits data two ways at every step, works for both categories and numbers, and then trims the tree back so it does not memorize noise.
- information gaininformation gain, n.: How much a question reduces uncertainty about the answer. A decision tree asks first the question that removes the most uncertainty, measured in bits.
- entropyentropy, n.: A measure of unpredictability. A fair coin toss has more entropy than a rigged one; an AI that is unsure of its next word has high entropy.
- feature importancefeature importance, n.: A score for how much each input — age, income, location — matters to a model's predictions. It helps people understand, debug and simplify models, but can mislead if read as cause and effect.
- random forestrandom forest, n.: A prediction method that builds hundreds of decision trees, each from a random sample of the data and features, and lets them vote. The crowd is far more reliable than any single tree.
- information gaininformation gain, n.: How much a question reduces uncertainty about the answer. A decision tree asks first the question that removes the most uncertainty, measured in bits.
Mentioned in
decision treedecision tree, n.: A model that makes a prediction by asking a series of yes-or-no questions about the input, like a flowchart, with each answer leading to the next question until it reaches a final answer., feature importancefeature importance, n.: A score for how much each input — age, income, location — matters to a model's predictions. It helps people understand, debug and simplify models, but can mislead if read as cause and effect., information gaininformation gain, n.: How much a question reduces uncertainty about the answer. A decision tree asks first the question that removes the most uncertainty, measured in bits., CARTCART, abbr.: A classic recipe for building decision trees that splits data two ways at every step, works for both categories and numbers, and then trims the tree back so it does not memorize noise.