information retrieval
In plain wordsThe study and engineering of search: how to organize huge piles of documents so that the few that matter to someone's question can be found and ranked quickly. AI systems rely on it to look things up.
The field that builds and studies search systems: methods for indexing large collections of mostly unstructured material (text, and increasingly images, audio and code), matching them against what a user is looking for, usually stated as a query, and ranking the results by likely usefulness. Its core components are indexing (for example the inverted indexinverted index, n.: A lookup table that lists, for every word, all the documents containing it — like the index at the back of a book — so a search engine can find matches almost instantly.), retrieval models that score query–document relevance (the vector space modelvector space model, n.: An early, still-influential idea for search: treat every document and query as an arrow in a space with one direction per word, and rank documents by how closely their arrows point the same way as the query's., probabilistic models such as BM25BM25, prop. n.: A decades-old formula that search engines use to score how well a document matches a query, rewarding rare query words and repeated matches — with diminishing returns — and correcting for long documents., learning to ranklearning to rank, n.: Training a model to put items in the best order — search results, products, recommendations — by learning from examples of which items people judged more relevant or chose more often. and neural dense retrievaldense retrieval, n.: Finding relevant documents by turning the question and every document into lists of numbers that capture meaning, then picking the documents whose numbers are closest to the question's.), and evaluation against human relevance judgments with measures such as precisionprecision, n.: Of all the items a model flagged as positive, the share that really were positive; a spam filter with high precision rarely sends genuine emails to the spam folder., recallrecall, n.: Of all the items that really were positive, the share the model managed to find; a cancer screen with high recall misses very few people who actually have cancer., mean reciprocal rankmean reciprocal rank, n.: A score for search: for each question, note how far down the first correct answer appears, take one over that position, and average across questions. First place scores 1, second place 0.5. and nDCGnormalized discounted cumulative gain, n.: A score for a ranked list of results that rewards putting the most relevant items near the top, scaled so that a perfect ordering scores 1.. Unlike database querying, IR deals with vaguely stated needs and graded, partly subjective relevance. Web search, enterprise search, recommendationrecommender system, n.: Software that guesses which songs, products, posts or jobs you are most likely to want and shows those first, as when a video app lines up the next clip or a shop suggests an add-on. and the retrieval stage of retrieval-augmented generationretrieval-augmented generation, n.: Giving an AI an open-book exam: before answering, it looks up relevant passages in a set of documents and bases its reply on them, often citing where each point came from. are all IR applications.
The RAG team hired someone with an information retrieval background, and recall jumped once she fixed the tokenization.
Origin
Calvin Mooers coined the term in 1950, in a paper presented at an Association for Computing Machinery meeting and in bulletins of his Zator Company, for the task of finding stored information whose location, or even existence, the searcher does not know in advance. Gerard Salton's SMART system (from the 1960s), Cyril Cleverdon's Cranfield experiments and NIST's Text REtrieval Conference (TREC, from 1992) shaped the field.
Usage note
Retrieval is judged by relevance to a need, not by exact match to a query, which is why IR evaluation relies on human judgments and rank-sensitive metrics.
Related
- BM25BM25, prop. n.: A decades-old formula that search engines use to score how well a document matches a query, rewarding rare query words and repeated matches — with diminishing returns — and correcting for long documents.
- inverted indexinverted index, n.: A lookup table that lists, for every word, all the documents containing it — like the index at the back of a book — so a search engine can find matches almost instantly.
- vector space modelvector space model, n.: An early, still-influential idea for search: treat every document and query as an arrow in a space with one direction per word, and rank documents by how closely their arrows point the same way as the query's.
- dense retrievaldense retrieval, n.: Finding relevant documents by turning the question and every document into lists of numbers that capture meaning, then picking the documents whose numbers are closest to the question's.
- retrieval-augmented generationretrieval-augmented generation, n.: Giving an AI an open-book exam: before answering, it looks up relevant passages in a set of documents and bases its reply on them, often citing where each point came from.
- semantic searchsemantic search, n.: Search that understands what you mean, not just the words you typed, so a search for 'cheap places to stay' also finds 'budget hotels'.
- learning to ranklearning to rank, n.: Training a model to put items in the best order — search results, products, recommendations — by learning from examples of which items people judged more relevant or chose more often.
- normalized discounted cumulative gainnormalized discounted cumulative gain, n.: A score for a ranked list of results that rewards putting the most relevant items near the top, scaled so that a perfect ordering scores 1.
Sources
- Mooers, C. N. (1950). The theory of digital handling of non-numerical information and its implications to machine economics. Paper presented at the Association for Computing Machinery meeting, Rutgers University, March 1950.
- Manning, C. D., Raghavan, P. & Schütze, H. (2008). Introduction to Information Retrieval. Cambridge University Press.
Mentioned in
natural language processingnatural language processing, n.: The field that teaches computers to handle human language — to read, interpret, translate, summarize and produce text and speech. It is one of the oldest and largest areas of artificial intelligence., retrieval-augmented generationretrieval-augmented generation, n.: Giving an AI an open-book exam: before answering, it looks up relevant passages in a set of documents and bases its reply on them, often citing where each point came from., bag of wordsbag of words, n.: A simple way of turning text into numbers by counting how often each word appears and ignoring word order — like describing a recipe only by its shopping list., BM25BM25, prop. n.: A decades-old formula that search engines use to score how well a document matches a query, rewarding rare query words and repeated matches — with diminishing returns — and correcting for long documents., content-based filteringcontent-based filtering, n.: Recommending items that resemble what you already liked, judged by the items' own features, such as a film's genre, cast and plot, rather than by other people's choices., generative engine optimizationgenerative engine optimization, n.: Shaping web content so that AI chatbots and AI search features are more likely to find it, quote it and recommend it — the AI-era version of search engine optimization., inverted indexinverted index, n.: A lookup table that lists, for every word, all the documents containing it — like the index at the back of a book — so a search engine can find matches almost instantly., mean average precisionmean average precision, n.: The standard score for object detectors: it rewards finding all the objects while raising few false alarms, averaged over every type of object. Higher is better, up to 100., mean reciprocal rankmean reciprocal rank, n.: A score for search: for each question, note how far down the first correct answer appears, take one over that position, and average across questions. First place scores 1, second place 0.5., normalized discounted cumulative gainnormalized discounted cumulative gain, n.: A score for a ranked list of results that rewards putting the most relevant items near the top, scaled so that a perfect ordering scores 1., query rewritingquery rewriting, n.: Rephrasing a user's question before searching — fixing typos, adding missing details, splitting it into simpler questions — so the search engine finds better results., question answeringquestion answering, n.: Systems that reply to people's questions with the right answer, whether by finding it in a passage they are given, searching a database or the web, or drawing on what they already know., recall@krecall@k, n.: A score for search systems: of all the right answers that exist, what fraction showed up in the top k results? Higher means fewer relevant items were missed., sparse retrievalsparse retrieval, n.: Finding documents by the words they contain, giving more weight to rare, telling words — the classic way search engines work., TF-IDFTF-IDF, abbr.: A simple score for picking out the words that characterize a document: a word counts for more the more often it appears there, and for less the more of the collection's other documents also use it., TRECTREC, prop. n.: A yearly contest, run by a US government standards agency since 1992, in which search-engine researchers test their systems on the same documents and questions, with people judging which results are actually relevant., web crawlerweb crawler, n.: A program that automatically travels the web, following links from page to page and downloading what it finds — used by search engines, archives and AI companies to gather data., latent semantic analysislatent semantic analysis, n.: An early technique for capturing word meaning by counting which words appear in which documents, then compressing those counts so that words used in similar contexts end up close together., learning to ranklearning to rank, n.: Training a model to put items in the best order — search results, products, recommendations — by learning from examples of which items people judged more relevant or chose more often., MS MARCOMS MARCO, prop. n.: A large Microsoft dataset of real Bing search questions paired with web passages that answer them, used to train and test most modern AI search models., stemmingstemming, n.: Cutting words down to a common root so that variants count as the same word — 'connected', 'connecting' and 'connection' all become 'connect' — to help search and text analysis., vector space modelvector space model, n.: An early, still-influential idea for search: treat every document and query as an arrow in a space with one direction per word, and rank documents by how closely their arrows point the same way as the query's., word sense disambiguationword sense disambiguation, n.: Working out which meaning of a word is intended in context — whether 'bank' means a riverside or a place that keeps money.