TREC
In plain wordsA yearly contest, run by a US government standards agency since 1992, in which search-engine researchers test their systems on the same documents and questions, with people judging which results are actually relevant.
An annual evaluation campaign for information retrievalinformation retrieval, n.: The study and engineering of search: how to organize huge piles of documents so that the few that matter to someone's question can be found and ranked quickly. AI systems rely on it to look things up., run since 1992 by the US National Institute of Standards and Technology (NIST), that supplies shared document collections, topics (statements of an information need) and human relevance judgments. Work is organized into tracks that change over the years: ad hoc and routing search, question answeringquestion answering, n.: Systems that reply to people's questions with the right answer, whether by finding it in a passage they are given, searching a database or the web, or drawing on what they already know., web, spam, legal discovery, genomics, microblogs, clinical decision support, conversational search, deep-learning ranking on MS MARCOMS MARCO, prop. n.: A large Microsoft dataset of real Bing search questions paired with web passages that answer them, used to train and test most modern AI search models. and, most recently, retrieval-augmented generationretrieval-augmented generation, n.: Giving an AI an open-book exam: before answering, it looks up relevant passages in a set of documents and bases its reply on them, often citing where each point came from.. Because judging every document is impossible, TREC popularized pooling, in which assessors judge only the union of the top-ranked documents from all submitted runs. Its reusable test collections, metrics such as mean average precisionmean average precision, n.: The standard score for object detectors: it rewards finding all the objects while raising few false alarms, averaged over every type of object. Higher is better, up to 100. and the trec_eval scoring tool scaled the older Cranfield laboratory method to gigabyte- and later terabyte-sized collections, and its model was copied by CLEF in Europe, NTCIR in Japan and FIRE in India.
We report nDCG@10 on the TREC Deep Learning 2019 and 2020 passage-ranking tasks.
Origin
Launched in 1992 by NIST with DARPA co-sponsorship as part of the TIPSTER text program, under Donna Harman, who led it in its early years; Ellen Voorhees later managed it for many years. The first meeting, TREC-1, was held in November 1992.
Usage note
Pooled judgments are incomplete, so systems unlike those that formed the original pool, such as neural rankers evaluated on old collections, can be scored unfairly; this pooling bias is a recurring methodological concern. TREC is a workshop built around shared evaluation, not a peer-reviewed venue for new methods.
Related
- information retrievalinformation retrieval, n.: The study and engineering of search: how to organize huge piles of documents so that the few that matter to someone's question can be found and ranked quickly. AI systems rely on it to look things up.
- Message Understanding ConferenceMessage Understanding Conference, prop. n.: A series of US government-funded contests held from 1987 to 1998 in which research teams built programs to pull key facts, such as who did what, where and when, out of news and military messages.
- mean average precisionmean average precision, n.: The standard score for object detectors: it rewards finding all the objects while raising few false alarms, averaged over every type of object. Higher is better, up to 100.
- MS MARCOMS MARCO, prop. n.: A large Microsoft dataset of real Bing search questions paired with web passages that answer them, used to train and test most modern AI search models.
- BEIRBEIR, prop. n.: A collection of 18 different search tests — on topics from medicine to finance to fact-checking — used to see whether a search model works on kinds of data it was never trained on.
- BM25BM25, prop. n.: A decades-old formula that search engines use to score how well a document matches a query, rewarding rare query words and repeated matches — with diminishing returns — and correcting for long documents.
- benchmarkbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time.
Sources
- Voorhees, E. M. & Harman, D. K. (Eds.) (2005). TREC: Experiment and Evaluation in Information Retrieval. MIT Press.
Mentioned in
Message Understanding ConferenceMessage Understanding Conference, prop. n.: A series of US government-funded contests held from 1987 to 1998 in which research teams built programs to pull key facts, such as who did what, where and when, out of news and military messages.