cross-encoder
In plain wordsAn AI model that reads a question and a candidate passage together, side by side, and outputs one relevance score. Accurate but slow, so it is used to reorder a short list.
A model that scores a pair of texts — typically a query and a candidate document — by feeding them jointly into one transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it. encoder, separated by a special token, so that self-attentionself-attention, n.: Attention in which a passage looks at itself: every word checks every other word in the same text to work out its meaning in context, such as which noun 'it' refers to. lets every token of one text attend to every token of the other, and mapping the joint representation to a relevance or similarity score. This early, full interaction makes cross-encoders the most accurate common architecture for rerankingreranking, n.: A second, more careful pass over search results: a fast method grabs a few hundred candidates, then a slower, smarter model reorders them so the best come first. and for sentence-pair tasks such as paraphrase detection and natural language inference. But nothing can be precomputed: every query–document pair needs its own forward passforward pass, n.: The step where a neural network takes an input and runs it through its layers, one after another, to produce an output — its answer or prediction., so scoring millions of documents per query is impractical. Systems therefore shortlist candidates with a fast bi-encoderbi-encoder, n.: An AI design that turns a question and a document into number lists separately, so documents can be processed once in advance and compared with any new question in a flash. or BM25BM25, prop. n.: A decades-old formula that search engines use to score how well a document matches a query, rewarding rare query words and repeated matches — with diminishing returns — and correcting for long documents. and reorder them with a cross-encoder, or distill cross-encoder judgments into bi-encoders.
The cross-encoder takes 40 ms per pair, so we only run it on the top 50 hits.
Origin
Rodrigo Nogueira and Kyunghyun Cho's BERT passage reranker (2019) is the classic example. The name, paired with 'bi-encoder', appears in the Poly-encoders paper by Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux and Jason Weston (Facebook AI Research, April 2019) and spread through Nils Reimers's Sentence-Transformers library.
Related
- bi-encoderbi-encoder, n.: An AI design that turns a question and a document into number lists separately, so documents can be processed once in advance and compared with any new question in a flash.
- rerankingreranking, n.: A second, more careful pass over search results: a fast method grabs a few hundred candidates, then a slower, smarter model reorders them so the best come first.
- late interactionlate interaction, n.: A search design that keeps a separate number-list for every word of a document and compares them with the question's words at the last moment — more precise than one list per document, yet still fast.
- ColBERTColBERT, prop. n.: A search model that compares a question and a document word by word — matching each question word to its best partner in the document — while still letting documents be processed ahead of time for speed.
- BERTBERT, prop. n.: An influential 2018 Google language model that learned to understand text by filling in hidden words. It made search engines and many language tools much better at grasping meaning in context.
- self-attentionself-attention, n.: Attention in which a passage looks at itself: every word checks every other word in the same text to work out its meaning in context, such as which noun 'it' refers to.
- bi-encoderbi-encoder, n.: An AI design that turns a question and a document into number lists separately, so documents can be processed once in advance and compared with any new question in a flash.
Sources
- Humeau, S., Shuster, K., Lachaux, M.-A. & Weston, J. (2019). Poly-encoders: Transformer Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring. arXiv:1905.01969. arxiv.org/abs/1905.01969
- Nogueira, R. & Cho, K. (2019). Passage Re-ranking with BERT. arXiv:1901.04085. arxiv.org/abs/1901.04085
Mentioned in
bi-encoderbi-encoder, n.: An AI design that turns a question and a document into number lists separately, so documents can be processed once in advance and compared with any new question in a flash., encoder-only transformerencoder-only transformer, n.: A transformer that reads a whole text at once, in both directions, to understand it rather than to write new text. Such models power search, sorting documents by topic and spotting names in text., rerankingreranking, n.: A second, more careful pass over search results: a fast method grabs a few hundred candidates, then a slower, smarter model reorders them so the best come first., Sentence-BERTSentence-BERT, prop. n.: A way of turning a whole sentence into a list of numbers so that sentences with similar meanings get similar numbers, which makes it fast to search millions of texts by meaning rather than by exact words., two-tower modeltwo-tower model, n.: A design for recommendation and search in which one neural network turns users or queries into lists of numbers and another does the same for items, so good matches can be found by quick comparison., ColBERTColBERT, prop. n.: A search model that compares a question and a document word by word — matching each question word to its best partner in the document — while still letting documents be processed ahead of time for speed., late interactionlate interaction, n.: A search design that keeps a separate number-list for every word of a document and compares them with the question's words at the last moment — more precise than one list per document, yet still fast., learning to ranklearning to rank, n.: Training a model to put items in the best order — search results, products, recommendations — by learning from examples of which items people judged more relevant or chose more often., MS MARCOMS MARCO, prop. n.: A large Microsoft dataset of real Bing search questions paired with web passages that answer them, used to train and test most modern AI search models., semantic textual similaritysemantic textual similarity, n.: How close two sentences are in meaning, scored from 'about different things' to 'say exactly the same thing' — and the tests that check whether a computer's scores match people's judgments.