mean reciprocal rank
In plain wordsA score for search: for each question, note how far down the first correct answer appears, take one over that position, and average across questions. First place scores 1, second place 0.5.
An evaluation measure for ranked results that averages, over a set of queries, the reciprocal 1/r of the position r of the first relevant result (a first hit at rank 4 earns 0.25), counting 0 when nothing relevant is found (or nothing within a cutoff k, as in MRR@10). MRR suits tasks where the user needs one good answer — question answering, known-item search, autocomplete — and is easy to interpret, but it ignores every relevant result after the first and does not use graded relevance; nDCGnormalized discounted cumulative gain, n.: A score for a ranked list of results that rewards putting the most relevant items near the top, scaled so that a perfect ordering scores 1. or mean average precision are preferred when several results matter. MRR@10 is the official metric of the MS MARCOMS MARCO, prop. n.: A large Microsoft dataset of real Bing search questions paired with web passages that answer them, used to train and test most modern AI search models. passage-ranking task.
MRR@10 went from 0.33 to 0.39 after fine-tuning, so the right passage is moving toward the top.
Origin
Reciprocal-rank scoring was the official measure of the TREC-8 Question Answering track (1999), described by Ellen Voorhees, and spread to passage ranking, recommendation and dialogue-response selection; the MS MARCO passage-ranking task made MRR@10 a standard neural-retrieval metric.
Usage note
Not to be confused with reciprocal rank fusionreciprocal rank fusion, n.: A simple recipe for merging several ranked lists of search results: each result earns points according to how high it ranks in each list, and the totals decide the final order., which combines rankings rather than evaluating them.
Related
- normalized discounted cumulative gainnormalized discounted cumulative gain, n.: A score for a ranked list of results that rewards putting the most relevant items near the top, scaled so that a perfect ordering scores 1.
- recall@krecall@k, n.: A score for search systems: of all the right answers that exist, what fraction showed up in the top k results? Higher means fewer relevant items were missed.
- rerankingreranking, n.: A second, more careful pass over search results: a fast method grabs a few hundred candidates, then a slower, smarter model reorders them so the best come first.
- information retrievalinformation retrieval, n.: The study and engineering of search: how to organize huge piles of documents so that the few that matter to someone's question can be found and ranked quickly. AI systems rely on it to look things up.
- question answeringquestion answering, n.: Systems that reply to people's questions with the right answer, whether by finding it in a passage they are given, searching a database or the web, or drawing on what they already know.
- evaluationevaluation, n.: Testing an AI system to find out what it can and cannot do, how well, and how safely — from exam-style quizzes to hands-on trials and expert reviews.
- reciprocal rank fusionreciprocal rank fusion, n.: A simple recipe for merging several ranked lists of search results: each result earns points according to how high it ranks in each list, and the totals decide the final order.
Sources
- Voorhees, E. M. (1999). The TREC-8 Question Answering Track Report. Proceedings of the Eighth Text REtrieval Conference (TREC-8), NIST.
Mentioned in
information retrievalinformation retrieval, n.: The study and engineering of search: how to organize huge piles of documents so that the few that matter to someone's question can be found and ranked quickly. AI systems rely on it to look things up., normalized discounted cumulative gainnormalized discounted cumulative gain, n.: A score for a ranked list of results that rewards putting the most relevant items near the top, scaled so that a perfect ordering scores 1., recall@krecall@k, n.: A score for search systems: of all the right answers that exist, what fraction showed up in the top k results? Higher means fewer relevant items were missed., MS MARCOMS MARCO, prop. n.: A large Microsoft dataset of real Bing search questions paired with web passages that answer them, used to train and test most modern AI search models., reciprocal rank fusionreciprocal rank fusion, n.: A simple recipe for merging several ranked lists of search results: each result earns points according to how high it ranks in each list, and the totals decide the final order.