MS MARCO
In plain wordsA large Microsoft dataset of real Bing search questions paired with web passages that answer them, used to train and test most modern AI search models.
A large-scale dataset built from anonymized questions submitted to the Bing search engine, each paired with passages extracted from retrieved web pages and human annotations of which passages answer it. First released for machine reading comprehension, it is best known for its passage-ranking task — roughly 8.8 million passages and several hundred thousand training queries with sparse relevance labels — scored by MRR@10mean reciprocal rank, n.: A score for search: for each question, note how far down the first correct answer appears, take one over that position, and average across questions. First place scores 1, second place 0.5.. As the first public collection large enough to train neural rankers, it became the default training set for cross-encodercross-encoder, n.: An AI model that reads a question and a candidate passage together, side by side, and outputs one relevance score. Accurate but slow, so it is used to reorder a short list. rerankers, dense retrieversdense retrieval, n.: Finding relevant documents by turning the question and every document into lists of numbers that capture meaning, then picking the documents whose numbers are closest to the question's. and ColBERTColBERT, prop. n.: A search model that compares a question and a document word by word — matching each question word to its best partner in the document — while still letting documents be processed ahead of time for speed., and the basis of NIST's TREC Deep Learning track (2019–2023). Its limitations include shallow labels (often one known relevant passage per query, with other relevant passages unlabeled), English-only web text and licensing restricted to non-commercial research.
The retriever was fine-tuned on MS MARCO and then evaluated zero-shot on BEIR.
Origin
Released by Microsoft in 2016 and described by Tri Nguyen and colleagues in 'MS MARCO: A Human Generated MAchine Reading COmprehension Dataset'. Its passage-ranking leaderboard drew the first BERT rerankers in early 2019, and the TREC Deep Learning track used it from 2019 until the track's final edition in 2023.
Related
- mean reciprocal rankmean reciprocal rank, n.: A score for search: for each question, note how far down the first correct answer appears, take one over that position, and average across questions. First place scores 1, second place 0.5.
- BEIRBEIR, prop. n.: A collection of 18 different search tests — on topics from medicine to finance to fact-checking — used to see whether a search model works on kinds of data it was never trained on.
- dense retrievaldense retrieval, n.: Finding relevant documents by turning the question and every document into lists of numbers that capture meaning, then picking the documents whose numbers are closest to the question's.
- rerankingreranking, n.: A second, more careful pass over search results: a fast method grabs a few hundred candidates, then a slower, smarter model reorders them so the best come first.
- cross-encodercross-encoder, n.: An AI model that reads a question and a candidate passage together, side by side, and outputs one relevance score. Accurate but slow, so it is used to reorder a short list.
- information retrievalinformation retrieval, n.: The study and engineering of search: how to organize huge piles of documents so that the few that matter to someone's question can be found and ranked quickly. AI systems rely on it to look things up.
- benchmarkbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time.
Sources
- Nguyen, T. et al. (2016). MS MARCO: A Human Generated MAchine Reading COmprehension Dataset. arXiv:1611.09268; CoCo workshop at NIPS 2016. arxiv.org/abs/1611.09268
- Nogueira, R. & Cho, K. (2019). Passage Re-ranking with BERT. arXiv:1901.04085. arxiv.org/abs/1901.04085
Mentioned in
mean reciprocal rankmean reciprocal rank, n.: A score for search: for each question, note how far down the first correct answer appears, take one over that position, and average across questions. First place scores 1, second place 0.5., TRECTREC, prop. n.: A yearly contest, run by a US government standards agency since 1992, in which search-engine researchers test their systems on the same documents and questions, with people judging which results are actually relevant., BEIRBEIR, prop. n.: A collection of 18 different search tests — on topics from medicine to finance to fact-checking — used to see whether a search model works on kinds of data it was never trained on.