recall
In plain wordsOf all the items that really were positive, the share the model managed to find; a cancer screen with high recall misses very few people who actually have cancer.
- 1
Classification and retrieval
The fraction of actual positives that a model correctly identifies: TP/(TP + FN), where FN counts false negativesfalse negative, n.: A miss: a test or AI system gives the all-clear, for example no illness, no fraud or no danger, when the problem is in fact there.. In retrieval it is the fraction of all relevant documents that are returned (recall@krecall@k, n.: A score for search systems: of all the right answers that exist, what fraction showed up in the top k results? Higher means fewer relevant items were missed. for the top k results); in medicine and signal detection it is called sensitivity or the true positive rate, the y-axis of the ROC curveROC curve, n.: A graph showing how a detector's hit rate and false-alarm rate change together as it is made more or less trigger-happy; the closer the curve hugs the top-left corner, the better.. Recall answers 'of everything we should have caught, how much did we catch?' and matters most when misses are costly: disease screening, fraud detection, safety filtering, legal discovery. It trades off against precisionprecision, n.: Of all the items a model flagged as positive, the share that really were positive; a spam filter with high precision rarely sends genuine emails to the spam folder.: lowering the decision threshold catches more positives but admits more false positivesfalse positive, n.: A false alarm: a test or AI system raises a flag, for example for illness, fraud or danger, when nothing of the kind is really there., so recall alone can be trivially maximized by flagging everything. Unlike precision, recall does not depend on how common positives are, which keeps it stable across populations with different prevalence.
We tuned the triage model for 99% recall, accepting that nurses would review many false alarms.
- 2
Language models
The retrieval of stored information by a model: factual recall of knowledge acquired in pretraining (whose internal mechanisms interpretability research studies), verbatim recall of memorized training text (memorizationmemorization, n.: When an AI model stores specific pieces of its training data — a paragraph, a phone number, an image — closely enough to reproduce them, instead of only learning general patterns.), and recall of details from a long context windowcontext window, n.: How much an AI model can read and keep in view at once: the instructions, the conversation so far and any attached documents. Text that does not fit has to be cut, summarized or left out., as tested by needle in a haystackneedle in a haystack, n.: A test for AI models that read very long documents: one unrelated sentence is hidden somewhere in a mass of text, and the model is asked about it. evaluations.
Origin
The metric comes from information-retrieval research of the 1950s and 1960s, where work by Allen Kent and colleagues (1955) and Cyril Cleverdon's Cranfield tests established recall alongside precision. The synonym 'sensitivity' comes from medical diagnostic testing, where Jacob Yerushalmy defined sensitivity and specificity in 1947.
Related
- precisionprecision, n.: Of all the items a model flagged as positive, the share that really were positive; a spam filter with high precision rarely sends genuine emails to the spam folder.
- F1 scoreF1 score, n.: A single score that combines how many of a model's alarms are right and how many real cases it catches, and stays low unless both are good.
- precision–recall curveprecision–recall curve, n.: A graph showing how a model's share of correct alarms and its share of cases caught trade off as it is made more or less cautious; especially useful when positive cases are rare.
- ROC curveROC curve, n.: A graph showing how a detector's hit rate and false-alarm rate change together as it is made more or less trigger-happy; the closer the curve hugs the top-left corner, the better.
- false negativefalse negative, n.: A miss: a test or AI system gives the all-clear, for example no illness, no fraud or no danger, when the problem is in fact there.
- specificityspecificity, n.: Of all the cases that really were negative, the share a test correctly cleared; a highly specific test rarely raises false alarms on healthy people.
- recall@krecall@k, n.: A score for search systems: of all the right answers that exist, what fraction showed up in the top k results? Higher means fewer relevant items were missed.
- confusion matrixconfusion matrix, n.: A table that counts a model's right and wrong answers by type, showing for each true category how often each category was predicted, so you can see exactly which things it mixes up.
- precisionprecision, n.: Of all the items a model flagged as positive, the share that really were positive; a spam filter with high precision rarely sends genuine emails to the spam folder.
Sources
- Kent, A., Berry, M. M., Luehrs, F. U. & Perry, J. W. (1955). Machine Literature Searching VIII. Operational Criteria for Designing Information Retrieval Systems. American Documentation 6(2).
- Yerushalmy, J. (1947). Statistical Problems in Assessing Methods of Medical Diagnosis, with Special Reference to X-Ray Techniques. Public Health Reports 62(40).
Mentioned in
accuracyaccuracy, n.: The share of a model's answers that are right, such as 95 out of 100. Simple and intuitive, but misleading when one kind of answer is far more common than the others., classificationclassification, n.: Sorting things into categories — spam or not spam, which animal is in a photo, which language a text is written in — by learning from examples that are already labeled., AUCAUC, abbr.: A single number from 0 to 1 summarizing how well a model ranks real positive cases above negative ones: 0.5 is no better than a coin flip and 1.0 is perfect., binary classificationbinary classification, n.: Sorting things into exactly two groups, such as spam versus not spam or sick versus healthy. Most yes-or-no decisions made by AI systems are of this kind., class imbalanceclass imbalance, n.: When one category is far rarer than the others in the data — say, one fraud case per thousand transactions — so a model can look accurate while missing nearly every rare case that matters., confusion matrixconfusion matrix, n.: A table that counts a model's right and wrong answers by type, showing for each true category how often each category was predicted, so you can see exactly which things it mixes up., decision thresholddecision threshold, n.: The cut-off score above which a model's prediction counts as 'yes'; moving it trades missed cases against false alarms, like setting how sensitive a smoke alarm should be., F1 scoreF1 score, n.: A single score that combines how many of a model's alarms are right and how many real cases it catches, and stays low unless both are good., false negativefalse negative, n.: A miss: a test or AI system gives the all-clear, for example no illness, no fraud or no danger, when the problem is in fact there., information retrievalinformation retrieval, n.: The study and engineering of search: how to organize huge piles of documents so that the few that matter to someone's question can be found and ranked quickly. AI systems rely on it to look things up., mean average precisionmean average precision, n.: The standard score for object detectors: it rewards finding all the objects while raising few false alarms, averaged over every type of object. Higher is better, up to 100., multilabel classificationmultilabel classification, n.: Tagging each item with any number of labels at once — a photo can show a beach, a dog and a sunset — instead of picking just one category., named-entity recognitionnamed-entity recognition, n.: Software that spots names and similar expressions in text and labels each by type: in 'Maria flew from Lagos to Paris on Monday', it marks a person, two cities and a date., precisionprecision, n.: Of all the items a model flagged as positive, the share that really were positive; a spam filter with high precision rarely sends genuine emails to the spam folder., precision–recall curveprecision–recall curve, n.: A graph showing how a model's share of correct alarms and its share of cases caught trade off as it is made more or less cautious; especially useful when positive cases are rare., recall@krecall@k, n.: A score for search systems: of all the right answers that exist, what fraction showed up in the top k results? Higher means fewer relevant items were missed., ROC curveROC curve, n.: A graph showing how a detector's hit rate and false-alarm rate change together as it is made more or less trigger-happy; the closer the curve hugs the top-left corner, the better., specificityspecificity, n.: Of all the cases that really were negative, the share a test correctly cleared; a highly specific test rarely raises false alarms on healthy people., Message Understanding ConferenceMessage Understanding Conference, prop. n.: A series of US government-funded contests held from 1987 to 1998 in which research teams built programs to pull key facts, such as who did what, where and when, out of news and military messages.