AUC
In plain wordsA single number from 0 to 1 summarizing how well a model ranks real positive cases above negative ones: 0.5 is no better than a coin flip and 1.0 is perfect.
Short for area under the curve, almost always the area under the ROC curveROC curve, n.: A graph showing how a detector's hit rate and false-alarm rate change together as it is made more or less trigger-happy; the closer the curve hugs the top-left corner, the better. (AUROC or ROC AUC), which medicine calls the c-statistic. Read probabilistically, it is the chance that the classifier's scores put a positive and a negative example, each drawn at random, in the right order, which makes it the normalized Mann–Whitney U statistic. An AUC of 0.5 means ranking no better than chance, 1.0 perfect separation, and values below 0.5 a systematically inverted ranking. AUC is threshold-free and insensitive to class prevalence and to monotone transformations of scores, which makes it a popular single number for comparing models. Those same properties are its limits: it ignores calibrationcalibration, n.: How well a model's stated confidence matches reality: when a well-calibrated system says it is 80% sure, it turns out to be right about 80% of the time., weighs all thresholds equally, including irrelevant ones, and can look excellent under heavy class imbalanceclass imbalance, n.: When one category is far rarer than the others in the data — say, one fraud case per thousand transactions — so a model can look accurate while missing nearly every rare case that matters. while precision at useful thresholds is poor, so the area under the precision–recall curveprecision–recall curve, n.: A graph showing how a model's share of correct alarms and its share of cases caught trade off as it is made more or less cautious; especially useful when positive cases are rare. (AUPRC, closely related to average precision) is often reported alongside it. Two models with equal AUC can have crossing ROC curves and suit different operating points.
The new model raised AUC from 0.86 to 0.89, but at our operating threshold precision barely moved.
Origin
In psychophysics, David Green and John Swets (1966) showed that the area equals accuracy in a two-alternative forced-choice task, and Donald Bamber (1975) related it to the Mann–Whitney U statistic; James Hanley and Barbara McNeil's 1982 Radiology paper 'The Meaning and Use of the Area under a Receiver Operating Characteristic (ROC) Curve' popularized the measure in medicine. Andrew Bradley's 1997 paper in Pattern Recognition helped make it a standard machine-learning metric.
Usage note
'AUC' alone usually means ROC AUC, but papers also use it for the precision–recall area and, in pharmacology, for the area under a drug-concentration curve, so check which is meant.
Related
- ROC curveROC curve, n.: A graph showing how a detector's hit rate and false-alarm rate change together as it is made more or less trigger-happy; the closer the curve hugs the top-left corner, the better.
- precision–recall curveprecision–recall curve, n.: A graph showing how a model's share of correct alarms and its share of cases caught trade off as it is made more or less cautious; especially useful when positive cases are rare.
- recallrecall, n.: Of all the items that really were positive, the share the model managed to find; a cancer screen with high recall misses very few people who actually have cancer.
- specificityspecificity, n.: Of all the cases that really were negative, the share a test correctly cleared; a highly specific test rarely raises false alarms on healthy people.
- calibrationcalibration, n.: How well a model's stated confidence matches reality: when a well-calibrated system says it is 80% sure, it turns out to be right about 80% of the time.
- class imbalanceclass imbalance, n.: When one category is far rarer than the others in the data — say, one fraud case per thousand transactions — so a model can look accurate while missing nearly every rare case that matters.
- learning to ranklearning to rank, n.: Training a model to put items in the best order — search results, products, recommendations — by learning from examples of which items people judged more relevant or chose more often.
Sources
- Hanley, J. A. & McNeil, B. J. (1982). The Meaning and Use of the Area under a Receiver Operating Characteristic (ROC) Curve. Radiology 143(1).
- Bradley, A. P. (1997). The Use of the Area under the ROC Curve in the Evaluation of Machine Learning Algorithms. Pattern Recognition 30(7).
- Bamber, D. (1975). The Area above the Ordinal Dominance Graph and the Area below the Receiver Operating Characteristic Graph. Journal of Mathematical Psychology 12(4).
Mentioned in
bootstrapbootstrap, n.: Estimating how much a result might vary by repeatedly re-drawing samples, with replacement, from the data you already have — like re-running a study many times without collecting new data., calibrationcalibration, n.: How well a model's stated confidence matches reality: when a well-calibrated system says it is 80% sure, it turns out to be right about 80% of the time., click-through rate predictionclick-through rate prediction, n.: Estimating how likely a particular person is to click on a particular ad, search result or recommended item: a prediction made billions of times a day to decide what each user is shown., Gini impurityGini impurity, n.: A score for how mixed a group is: zero when every item belongs to the same category, higher when categories are jumbled together. Decision trees choose splits that make groups purer., precision–recall curveprecision–recall curve, n.: A graph showing how a model's share of correct alarms and its share of cases caught trade off as it is made more or less cautious; especially useful when positive cases are rare., ROC curveROC curve, n.: A graph showing how a detector's hit rate and false-alarm rate change together as it is made more or less trigger-happy; the closer the curve hugs the top-left corner, the better., specificityspecificity, n.: Of all the cases that really were negative, the share a test correctly cleared; a highly specific test rarely raises false alarms on healthy people., proper scoring ruleproper scoring rule, n.: A way of grading probability forecasts that gives the best expected score only to a forecaster who reports what they really believe, so there is nothing to gain by exaggerating or hedging.