confusion matrix
In plain wordsA table that counts a model's right and wrong answers by type, showing for each true category how often each category was predicted, so you can see exactly which things it mixes up.
A tally of a classifier's outcomes on labeled data, laid out as a K × K grid for K classes, in which each cell counts the examples of one true class that received one particular predicted label. With true classes as rows and predictions as columns (some tools transpose this), correct answers accumulate along the main diagonal, while each off-diagonal count records how often one class was mistaken for a specific other. For binary classificationbinary classification, n.: Sorting things into exactly two groups, such as spam versus not spam or sick versus healthy. Most yes-or-no decisions made by AI systems are of this kind. it reduces to four counts, true positives (TP), false positivesfalse positive, n.: A false alarm: a test or AI system raises a flag, for example for illness, fraud or danger, when nothing of the kind is really there. (FP), true negatives (TN) and false negativesfalse negative, n.: A miss: a test or AI system gives the all-clear, for example no illness, no fraud or no danger, when the problem is in fact there. (FN), from which most threshold-based metrics are computed: accuracyaccuracy, n.: The share of a model's answers that are right, such as 95 out of 100. Simple and intuitive, but misleading when one kind of answer is far more common than the others., precisionprecision, n.: Of all the items a model flagged as positive, the share that really were positive; a spam filter with high precision rarely sends genuine emails to the spam folder., recallrecall, n.: Of all the items that really were positive, the share the model managed to find; a cancer screen with high recall misses very few people who actually have cancer., specificityspecificity, n.: Of all the cases that really were negative, the share a test correctly cleared; a highly specific test rarely raises false alarms on healthy people., the F1 scoreF1 score, n.: A single score that combines how many of a model's alarms are right and how many real cases it catches, and stays low unless both are good. and the Matthews correlation coefficientMatthews correlation coefficient, n.: A single score from −1 to +1 for a yes-or-no classifier that uses all four kinds of right and wrong answers, so it stays honest even when one answer is far more common than the other.. Normalizing each row shows per-class recall; normalizing each column shows per-class precision. A confusion matrix describes one decision threshold; sweeping the threshold yields the ROC curveROC curve, n.: A graph showing how a detector's hit rate and false-alarm rate change together as it is made more or less trigger-happy; the closer the curve hugs the top-left corner, the better. and the precision–recall curveprecision–recall curve, n.: A graph showing how a model's share of correct alarms and its share of cases caught trade off as it is made more or less cautious; especially useful when positive cases are rare.. Inspecting it is the first step of error analysis, revealing systematic confusions such as a speech recognizer mixing up 'b' and 'p'.
The confusion matrix showed the errors were nearly all huskies labeled as wolves, never the reverse.
Origin
The table originated in human perceptual studies of auditory stimuli, where researchers recorded which sounds listeners confused with which, as in George Miller and Patricia Nicely's 1955 study of misheard English consonants. Early machine-learning researchers, including Frank Rosenblatt, used such matrices to compare human and machine classifications; remote sensing calls the same table an error matrix.
Related
- accuracyaccuracy, n.: The share of a model's answers that are right, such as 95 out of 100. Simple and intuitive, but misleading when one kind of answer is far more common than the others.
- precisionprecision, n.: Of all the items a model flagged as positive, the share that really were positive; a spam filter with high precision rarely sends genuine emails to the spam folder.
- recallrecall, n.: Of all the items that really were positive, the share the model managed to find; a cancer screen with high recall misses very few people who actually have cancer.
- false positivefalse positive, n.: A false alarm: a test or AI system raises a flag, for example for illness, fraud or danger, when nothing of the kind is really there.
- false negativefalse negative, n.: A miss: a test or AI system gives the all-clear, for example no illness, no fraud or no danger, when the problem is in fact there.
- specificityspecificity, n.: Of all the cases that really were negative, the share a test correctly cleared; a highly specific test rarely raises false alarms on healthy people.
- F1 scoreF1 score, n.: A single score that combines how many of a model's alarms are right and how many real cases it catches, and stays low unless both are good.
- ROC curveROC curve, n.: A graph showing how a detector's hit rate and false-alarm rate change together as it is made more or less trigger-happy; the closer the curve hugs the top-left corner, the better.
Sources
- Miller, G. A. & Nicely, P. E. (1955). An Analysis of Perceptual Confusions Among Some English Consonants. Journal of the Acoustical Society of America 27(2).
- Stehman, S. V. (1997). Selecting and Interpreting Measures of Thematic Classification Accuracy. Remote Sensing of Environment 62(1).
Mentioned in
accuracyaccuracy, n.: The share of a model's answers that are right, such as 95 out of 100. Simple and intuitive, but misleading when one kind of answer is far more common than the others., false positivefalse positive, n.: A false alarm: a test or AI system raises a flag, for example for illness, fraud or danger, when nothing of the kind is really there., binary classificationbinary classification, n.: Sorting things into exactly two groups, such as spam versus not spam or sick versus healthy. Most yes-or-no decisions made by AI systems are of this kind., decision thresholddecision threshold, n.: The cut-off score above which a model's prediction counts as 'yes'; moving it trades missed cases against false alarms, like setting how sensitive a smoke alarm should be., F1 scoreF1 score, n.: A single score that combines how many of a model's alarms are right and how many real cases it catches, and stays low unless both are good., false negativefalse negative, n.: A miss: a test or AI system gives the all-clear, for example no illness, no fraud or no danger, when the problem is in fact there., multiclass classificationmulticlass classification, n.: Sorting each item into exactly one of three or more categories — for example, deciding which of the ten digits a handwritten number shows., precisionprecision, n.: Of all the items a model flagged as positive, the share that really were positive; a spam filter with high precision rarely sends genuine emails to the spam folder., recallrecall, n.: Of all the items that really were positive, the share the model managed to find; a cancer screen with high recall misses very few people who actually have cancer., ROC curveROC curve, n.: A graph showing how a detector's hit rate and false-alarm rate change together as it is made more or less trigger-happy; the closer the curve hugs the top-left corner, the better., specificityspecificity, n.: Of all the cases that really were negative, the share a test correctly cleared; a highly specific test rarely raises false alarms on healthy people., Cohen's kappaCohen's kappa, n.: A score for how much two people labeling the same items agree beyond what lucky guessing would produce: 1 means perfect agreement, 0 means no better than chance., label shiftlabel shift, n.: A kind of mismatch where the mix of answers changes — say, a disease becomes far more common during an outbreak — while each individual case still looks the same as before., Matthews correlation coefficientMatthews correlation coefficient, n.: A single score from −1 to +1 for a yes-or-no classifier that uses all four kinds of right and wrong answers, so it stays honest even when one answer is far more common than the other.