ROC curve
In plain wordsA graph showing how a detector's hit rate and false-alarm rate change together as it is made more or less trigger-happy; the closer the curve hugs the top-left corner, the better.
A plot of a binary classifier's true positive rate (recallrecall, n.: Of all the items that really were positive, the share the model managed to find; a cancer screen with high recall misses very few people who actually have cancer., sensitivity) against its false positive rate (1 − specificityspecificity, n.: Of all the cases that really were negative, the share a test correctly cleared; a highly specific test rarely raises false alarms on healthy people.) as the decision threshold on its score moves from strictest to most lenient ('receiver operating characteristic'). Each threshold gives one point, and the curve runs from (0, 0), where nothing is flagged, to (1, 1), where everything is. A random classifier lies on the diagonal, a perfect one passes through the top-left corner, and a curve below the diagonal signals systematically inverted scores. ROC curves show the whole trade-off at once, are unaffected by class prevalence and by any monotone rescaling of scores, and let practitioners pick an operating point given the costs of each error; the area under the curveAUC, abbr.: A single number from 0 to 1 summarizing how well a model ranks real positive cases above negative ones: 0.5 is no better than a coin flip and 1.0 is perfect. summarizes them in one number. Under heavy class imbalanceclass imbalance, n.: When one category is far rarer than the others in the data — say, one fraud case per thousand transactions — so a model can look accurate while missing nearly every rare case that matters., however, a small false positive rate can still mean many false alarms, and the precision–recall curveprecision–recall curve, n.: A graph showing how a model's share of correct alarms and its share of cases caught trade off as it is made more or less cautious; especially useful when positive cases are rare. is often more informative. ROC analysis measures ranking quality, not calibrationcalibration, n.: How well a model's stated confidence matches reality: when a well-calibrated system says it is 80% sure, it turns out to be right about 80% of the time..
From the ROC curve we picked the threshold with a 2% false positive rate, which still caught 85% of the fraud.
Origin
Developed in signal detection theory for radar during and after the Second World War; the name refers to the operating characteristic of a radar receiver. Wilson Tanner, John Swets and colleagues brought it into psychophysics in the 1950s, Lee Lusted into medical decision making in the 1960s, and Kent Spackman into machine learning in 1989; Tom Fawcett's 2006 tutorial is a standard reference.
Related
- AUCAUC, abbr.: A single number from 0 to 1 summarizing how well a model ranks real positive cases above negative ones: 0.5 is no better than a coin flip and 1.0 is perfect.
- precision–recall curveprecision–recall curve, n.: A graph showing how a model's share of correct alarms and its share of cases caught trade off as it is made more or less cautious; especially useful when positive cases are rare.
- recallrecall, n.: Of all the items that really were positive, the share the model managed to find; a cancer screen with high recall misses very few people who actually have cancer.
- specificityspecificity, n.: Of all the cases that really were negative, the share a test correctly cleared; a highly specific test rarely raises false alarms on healthy people.
- false positivefalse positive, n.: A false alarm: a test or AI system raises a flag, for example for illness, fraud or danger, when nothing of the kind is really there.
- confusion matrixconfusion matrix, n.: A table that counts a model's right and wrong answers by type, showing for each true category how often each category was predicted, so you can see exactly which things it mixes up.
- class imbalanceclass imbalance, n.: When one category is far rarer than the others in the data — say, one fraud case per thousand transactions — so a model can look accurate while missing nearly every rare case that matters.
- precision–recall curveprecision–recall curve, n.: A graph showing how a model's share of correct alarms and its share of cases caught trade off as it is made more or less cautious; especially useful when positive cases are rare.
Sources
- Fawcett, T. (2006). An Introduction to ROC Analysis. Pattern Recognition Letters 27(8).
- Swets, J. A. (1988). Measuring the Accuracy of Diagnostic Systems. Science 240(4857).
- Spackman, K. A. (1989). Signal Detection Theory: Valuable Tools for Evaluating Inductive Learning. Proc. 6th International Workshop on Machine Learning.
Mentioned in
accuracyaccuracy, n.: The share of a model's answers that are right, such as 95 out of 100. Simple and intuitive, but misleading when one kind of answer is far more common than the others., classificationclassification, n.: Sorting things into categories — spam or not spam, which animal is in a photo, which language a text is written in — by learning from examples that are already labeled., false positivefalse positive, n.: A false alarm: a test or AI system raises a flag, for example for illness, fraud or danger, when nothing of the kind is really there., AUCAUC, abbr.: A single number from 0 to 1 summarizing how well a model ranks real positive cases above negative ones: 0.5 is no better than a coin flip and 1.0 is perfect., binary classificationbinary classification, n.: Sorting things into exactly two groups, such as spam versus not spam or sick versus healthy. Most yes-or-no decisions made by AI systems are of this kind., confusion matrixconfusion matrix, n.: A table that counts a model's right and wrong answers by type, showing for each true category how often each category was predicted, so you can see exactly which things it mixes up., decision thresholddecision threshold, n.: The cut-off score above which a model's prediction counts as 'yes'; moving it trades missed cases against false alarms, like setting how sensitive a smoke alarm should be., false negativefalse negative, n.: A miss: a test or AI system gives the all-clear, for example no illness, no fraud or no danger, when the problem is in fact there., precision–recall curveprecision–recall curve, n.: A graph showing how a model's share of correct alarms and its share of cases caught trade off as it is made more or less cautious; especially useful when positive cases are rare., recallrecall, n.: Of all the items that really were positive, the share the model managed to find; a cancer screen with high recall misses very few people who actually have cancer., specificityspecificity, n.: Of all the cases that really were negative, the share a test correctly cleared; a highly specific test rarely raises false alarms on healthy people.