hybrid search
In plain wordsSearching two ways at once — by exact keywords and by meaning — and merging the results, so you catch precise matches like product codes as well as differently worded matches.
Retrieval that runs sparsesparse retrieval, n.: Finding documents by the words they contain, giving more weight to rare, telling words — the classic way search engines work. (keyword, typically BM25BM25, prop. n.: A decades-old formula that search engines use to score how well a document matches a query, rewarding rare query words and repeated matches — with diminishing returns — and correcting for long documents.) and densedense retrieval, n.: Finding relevant documents by turning the question and every document into lists of numbers that capture meaning, then picking the documents whose numbers are closest to the question's. (embedding) search over the same collection and combines their results into one ranking. Combination is either by rank — most often reciprocal rank fusionreciprocal rank fusion, n.: A simple recipe for merging several ranked lists of search results: each result earns points according to how high it ranks in each list, and the totals decide the final order., which needs no score calibration — or by a weighted sum of normalized scores, with the weight tuned on labeled queries; some systems instead score one model's sparse and dense outputs together. The two signals fail in different ways: keywords catch exact names, codes and rare terms, while embeddings catch paraphrase and synonymy, so hybrids are usually more robust than either alone. Hybrid search followed by rerankingreranking, n.: A second, more careful pass over search results: a fast method grabs a few hundred candidates, then a slower, smarter model reorders them so the best come first. is a common default first stage for retrieval-augmented generationretrieval-augmented generation, n.: Giving an AI an open-book exam: before answering, it looks up relevant passages in a set of documents and bases its reply on them, often citing where each point came from..
Switching to hybrid search fixed queries for part numbers without hurting the natural-language ones.
Origin
Fusing the outputs of several retrieval systems ('data fusion') is long-standing IR practice, for example Edward Fox and Joseph Shaw's CombSUM and CombMNZ methods (TREC-2, 1993). The name 'hybrid search' for keyword-plus-vector retrieval spread in the early 2020s, as vector databases and search engines added both index types and RAG made the combination standard.
Related
- reciprocal rank fusionreciprocal rank fusion, n.: A simple recipe for merging several ranked lists of search results: each result earns points according to how high it ranks in each list, and the totals decide the final order.
- sparse retrievalsparse retrieval, n.: Finding documents by the words they contain, giving more weight to rare, telling words — the classic way search engines work.
- dense retrievaldense retrieval, n.: Finding relevant documents by turning the question and every document into lists of numbers that capture meaning, then picking the documents whose numbers are closest to the question's.
- BM25BM25, prop. n.: A decades-old formula that search engines use to score how well a document matches a query, rewarding rare query words and repeated matches — with diminishing returns — and correcting for long documents.
- rerankingreranking, n.: A second, more careful pass over search results: a fast method grabs a few hundred candidates, then a slower, smarter model reorders them so the best come first.
- vector databasevector database, n.: A database built to store embeddings — lists of numbers that represent meaning — and to quickly find the stored items most similar to a new one. It is the memory behind many AI search tools.
- retrieval-augmented generationretrieval-augmented generation, n.: Giving an AI an open-book exam: before answering, it looks up relevant passages in a set of documents and bases its reply on them, often citing where each point came from.
Sources
- Cormack, G. V., Clarke, C. L. A. & Büttcher, S. (2009). Reciprocal rank fusion outperforms Condorcet and individual rank learning methods. SIGIR 2009: 758–759. ir.webis.de/anthology/2009.sigirconf_conference-2009.146
Mentioned in
retrieval-augmented generationretrieval-augmented generation, n.: Giving an AI an open-book exam: before answering, it looks up relevant passages in a set of documents and bases its reply on them, often citing where each point came from., BM25BM25, prop. n.: A decades-old formula that search engines use to score how well a document matches a query, rewarding rare query words and repeated matches — with diminishing returns — and correcting for long documents., dense retrievaldense retrieval, n.: Finding relevant documents by turning the question and every document into lists of numbers that capture meaning, then picking the documents whose numbers are closest to the question's., inverted indexinverted index, n.: A lookup table that lists, for every word, all the documents containing it — like the index at the back of a book — so a search engine can find matches almost instantly., rerankingreranking, n.: A second, more careful pass over search results: a fast method grabs a few hundred candidates, then a slower, smarter model reorders them so the best come first., semantic searchsemantic search, n.: Search that understands what you mean, not just the words you typed, so a search for 'cheap places to stay' also finds 'budget hotels'., sparse retrievalsparse retrieval, n.: Finding documents by the words they contain, giving more weight to rare, telling words — the classic way search engines work., vector databasevector database, n.: A database built to store embeddings — lists of numbers that represent meaning — and to quickly find the stored items most similar to a new one. It is the memory behind many AI search tools., contextual retrievalcontextual retrieval, n.: A fix for searching cut-up documents: before storing each piece, an AI writes a short note saying where it came from and what it is about, so the piece still makes sense on its own., reciprocal rank fusionreciprocal rank fusion, n.: A simple recipe for merging several ranked lists of search results: each result earns points according to how high it ranks in each list, and the totals decide the final order.