Skip to content
Last Dictionary
Plane
AI Engineering & MLOpsEvaluation & BenchmarksMathematics & Statistics

normalized discounted cumulative gain

nounAlso: nDCG, NDCG, nDCG@k, discounted cumulative gain, DCG, cumulated gain

In plain wordsA score for a ranked list of results that rewards putting the most relevant items near the top, scaled so that a perfect ordering scores 1.

  1. A ranking-quality measure that uses graded relevance judgments (for example 0 for irrelevant up to 3 for perfect). Discounted cumulative gain (DCG) adds up the gain of each result in the top k, dividing by a logarithm of its rank so that relevant items found lower down count for less; nDCG divides this by the ideal DCG (IDCG) — the DCG of the best possible ordering of the judged items — giving a value between 0 and 1 that can be averaged across queries. Two gain conventions are common: the relevance grade itself, as in the original definition, or 2 raised to the grade minus 1, which emphasizes highly relevant results and is standard in web search and learning to ranklearning to rank, n.: Training a model to put items in the best order — search results, products, recommendations — by learning from examples of which items people judged more relevant or chose more often.. nDCG@10 is the headline metric of retrieval benchmarks such as BEIRBEIR, prop. n.: A collection of 18 different search tests — on topics from medicine to finance to fact-checking — used to see whether a search model works on kinds of data it was never trained on. and the retrieval tasks of MTEBMTEB, prop. n.: A large public test suite that scores AI models which turn text into lists of numbers on many different jobs, such as searching, grouping, classifying and matching translations, so people can compare them on one leaderboard..

    The reranker lifted nDCG@10 from 0.41 to 0.52 on our labeled queries.

DCG@k=∑i=1k2reli−1log⁡2(i+1),nDCG@k=DCG@kIDCG@k\mathrm{DCG@}k=\sum_{i=1}^{k}\frac{2^{rel_i}-1}{\log_2(i+1)},\qquad \mathrm{nDCG@}k=\frac{\mathrm{DCG@}k}{\mathrm{IDCG@}k}

Origin

Kalervo Järvelin and Jaana Kekäläinen of the University of Tampere introduced cumulated gain and discounted cumulated gain at SIGIR 2000 and the normalized form in ACM Transactions on Information Systems (October 2002). The exponential-gain variant spread through Chris Burges and colleagues' learning-to-rank work (2005) and web-search practice.

Usage note

The originators wrote 'cumulated'; 'cumulative' is now more common. Because conventions differ, nDCG values are comparable only when gain, discount and cutoff match.

Related

See also
  • learning to ranklearning to rank, n.: Training a model to put items in the best order — search results, products, recommendations — by learning from examples of which items people judged more relevant or chose more often.
  • mean reciprocal rankmean reciprocal rank, n.: A score for search: for each question, note how far down the first correct answer appears, take one over that position, and average across questions. First place scores 1, second place 0.5.
  • recall@krecall@k, n.: A score for search systems: of all the right answers that exist, what fraction showed up in the top k results? Higher means fewer relevant items were missed.
  • rerankingreranking, n.: A second, more careful pass over search results: a fast method grabs a few hundred candidates, then a slower, smarter model reorders them so the best come first.
  • information retrievalinformation retrieval, n.: The study and engineering of search: how to organize huge piles of documents so that the few that matter to someone's question can be found and ranked quickly. AI systems rely on it to look things up.
  • evaluationevaluation, n.: Testing an AI system to find out what it can and cannot do, how well, and how safely — from exam-style quizzes to hands-on trials and expert reviews.

Sources

  1. Järvelin, K. & Kekäläinen, J. (2000). IR evaluation methods for retrieving highly relevant documents. SIGIR 2000: 41–48. ir.webis.de/anthology/2000.sigirconf_conference-2000.7
  2. Järvelin, K. & Kekäläinen, J. (2002). Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems 20(4): 422–446. doi.org/10.1145/582415.582418

Mentioned in

recommender systemrecommender system, n.: Software that guesses which songs, products, posts or jobs you are most likely to want and shows those first, as when a video app lines up the next clip or a shop suggests an add-on., information retrievalinformation retrieval, n.: The study and engineering of search: how to organize huge piles of documents so that the few that matter to someone's question can be found and ranked quickly. AI systems rely on it to look things up., mean reciprocal rankmean reciprocal rank, n.: A score for search: for each question, note how far down the first correct answer appears, take one over that position, and average across questions. First place scores 1, second place 0.5., recall@krecall@k, n.: A score for search systems: of all the right answers that exist, what fraction showed up in the top k results? Higher means fewer relevant items were missed., rerankingreranking, n.: A second, more careful pass over search results: a fast method grabs a few hundred candidates, then a slower, smarter model reorders them so the best come first., BEIRBEIR, prop. n.: A collection of 18 different search tests — on topics from medicine to finance to fact-checking — used to see whether a search model works on kinds of data it was never trained on., learning to ranklearning to rank, n.: Training a model to put items in the best order — search results, products, recommendations — by learning from examples of which items people judged more relevant or chose more often.