precision–recall curve
In plain wordsA graph showing how a model's share of correct alarms and its share of cases caught trade off as it is made more or less cautious; especially useful when positive cases are rare.
A plot of precisionprecision, n.: Of all the items a model flagged as positive, the share that really were positive; a spam filter with high precision rarely sends genuine emails to the spam folder. (y-axis) against recallrecall, n.: Of all the items that really were positive, the share the model managed to find; a cancer screen with high recall misses very few people who actually have cancer. (x-axis) for a classifier or ranking system as its decision threshold varies. Lowering the threshold moves the curve rightward, catching more positives, usually at some cost in precision; the curve is jagged rather than monotone, because precision can rise or fall as each new item is admitted. A random classifier gives a horizontal line at a precision equal to the prevalence of positives, so the baseline depends on the data, unlike the fixed diagonal of the ROC curveROC curve, n.: A graph showing how a detector's hit rate and false-alarm rate change together as it is made more or less trigger-happy; the closer the curve hugs the top-left corner, the better.. Because precision registers false positives among many negatives, PR curves expose weaknesses under class imbalanceclass imbalance, n.: When one category is far rarer than the others in the data — say, one fraud case per thousand transactions — so a model can look accurate while missing nearly every rare case that matters. that ROC curves can hide, which makes them standard for rare-event detection, information retrieval and object detectionobject detection, n.: Finding every object of interest in a picture or video frame, saying what each one is and drawing a box around it, for example all the cars, cyclists and pedestrians in a street scene.. The area under the curve is usually estimated as average precision (AP), the mean of the precision values at the ranks where relevant items occur. Mean average precisionmean average precision, n.: The standard score for object detectors: it rewards finding all the objects while raising few false alarms, averaged over every type of object. Higher is better, up to 100. (mAP), the mean of AP over queries or classes, and in detection over IoUintersection over union, n.: A score from 0 to 1 for how well two shapes overlap, such as a predicted box and the true box around an object: the shared area divided by the combined area. thresholds, is the headline metric of benchmarks such as COCOCOCO, prop. n.: A widely used collection of everyday photos in which people outlined and named the objects, described each picture in sentences and marked body joints; it is used to train and compare vision systems..
With positives at 0.2% prevalence the ROC curve looked superb, but average precision was only 0.31.
Origin
Information-retrieval researchers plotted precision against recall from the Cranfield experiments of the 1960s onward, and average precision became a core measure of the Text REtrieval Conference (TREC) from 1992. In machine learning, Jesse Davis and Mark Goadrich's 2006 analysis of how PR and ROC curves relate popularized PR curves for imbalanced classification.
Usage note
Linear interpolation between PR points overestimates the area. Object-detection 'AP' has several incompatible definitions (11-point interpolated, all-point, COCO-style), so compare numbers only within one protocol.
Related
- precisionprecision, n.: Of all the items a model flagged as positive, the share that really were positive; a spam filter with high precision rarely sends genuine emails to the spam folder.
- recallrecall, n.: Of all the items that really were positive, the share the model managed to find; a cancer screen with high recall misses very few people who actually have cancer.
- ROC curveROC curve, n.: A graph showing how a detector's hit rate and false-alarm rate change together as it is made more or less trigger-happy; the closer the curve hugs the top-left corner, the better.
- AUCAUC, abbr.: A single number from 0 to 1 summarizing how well a model ranks real positive cases above negative ones: 0.5 is no better than a coin flip and 1.0 is perfect.
- class imbalanceclass imbalance, n.: When one category is far rarer than the others in the data — say, one fraud case per thousand transactions — so a model can look accurate while missing nearly every rare case that matters.
- F1 scoreF1 score, n.: A single score that combines how many of a model's alarms are right and how many real cases it catches, and stays low unless both are good.
- object detectionobject detection, n.: Finding every object of interest in a picture or video frame, saying what each one is and drawing a box around it, for example all the cars, cyclists and pedestrians in a street scene.
- ROC curveROC curve, n.: A graph showing how a detector's hit rate and false-alarm rate change together as it is made more or less trigger-happy; the closer the curve hugs the top-left corner, the better.
Sources
- Davis, J. & Goadrich, M. (2006). The Relationship Between Precision-Recall and ROC Curves. Proc. ICML 2006.
- Saito, T. & Rehmsmeier, M. (2015). The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets. PLOS ONE 10(3).
Mentioned in
AUCAUC, abbr.: A single number from 0 to 1 summarizing how well a model ranks real positive cases above negative ones: 0.5 is no better than a coin flip and 1.0 is perfect., binary classificationbinary classification, n.: Sorting things into exactly two groups, such as spam versus not spam or sick versus healthy. Most yes-or-no decisions made by AI systems are of this kind., class imbalanceclass imbalance, n.: When one category is far rarer than the others in the data — say, one fraud case per thousand transactions — so a model can look accurate while missing nearly every rare case that matters., confusion matrixconfusion matrix, n.: A table that counts a model's right and wrong answers by type, showing for each true category how often each category was predicted, so you can see exactly which things it mixes up., decision thresholddecision threshold, n.: The cut-off score above which a model's prediction counts as 'yes'; moving it trades missed cases against false alarms, like setting how sensitive a smoke alarm should be., F1 scoreF1 score, n.: A single score that combines how many of a model's alarms are right and how many real cases it catches, and stays low unless both are good., mean average precisionmean average precision, n.: The standard score for object detectors: it rewards finding all the objects while raising few false alarms, averaged over every type of object. Higher is better, up to 100., precisionprecision, n.: Of all the items a model flagged as positive, the share that really were positive; a spam filter with high precision rarely sends genuine emails to the spam folder., recallrecall, n.: Of all the items that really were positive, the share the model managed to find; a cancer screen with high recall misses very few people who actually have cancer., ROC curveROC curve, n.: A graph showing how a detector's hit rate and false-alarm rate change together as it is made more or less trigger-happy; the closer the curve hugs the top-left corner, the better., SMOTESMOTE, abbr.: A technique for lopsided data that invents extra examples of the rare category by blending each rare example with one of its nearest rare neighbors, so a model sees more of it during training.