bi-encoder
In plain wordsAn AI design that turns a question and a document into number lists separately, so documents can be processed once in advance and compared with any new question in a flash.
An architecture that encodes each of two inputs — such as a query and a document — independently into a fixed-size vector, usually with the same or twin transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it. encoders, and scores the pair with a cheap similarity function such as the dot productdot product, n.: A score of how well two lists of numbers line up, found by multiplying matching entries and adding the products. It is large when the lists point the same way and zero when they are at right angles. or cosine similaritycosine similarity, n.: A score from −1 to 1 for how closely two lists of numbers point the same way, ignoring their size. It is the standard test of whether two texts mean similar things.. Because document vectors do not depend on the query, they can be computed once and stored in an index, and a query can be compared against millions of them with vector searchvector search, n.: Searching by closeness instead of exact match: the query and all stored items are lists of numbers, and the search returns the items whose numbers are most similar to the query's.; this makes bi-encoders the standard model for dense retrievaldense retrieval, n.: Finding relevant documents by turning the question and every document into lists of numbers that capture meaning, then picking the documents whose numbers are closest to the question's. and text embeddingtext embedding, n.: A list of numbers that captures what a piece of text means, so that texts with similar meanings get similar lists. Computers use them to search, sort and group text by meaning.. The price is that the two texts never interact inside the network, so bi-encoders are less accurate than cross-encoderscross-encoder, n.: An AI model that reads a question and a candidate passage together, side by side, and outputs one relevance score. Accurate but slow, so it is used to reorder a short list. at fine relevance distinctions. They are trained with contrastive learningcontrastive learning, n.: A way to learn without labels by comparison: the model learns to pull together two versions of the same thing, such as two crops of one photo, and push apart unrelated things., often with hard negatives and distillation from a cross-encoder.
We embed the catalog nightly with the bi-encoder and encode only the query at search time.
Origin
Twin-network similarity learning dates to Siamese networksSiamese network, n.: A matching machine built from twin copies of one neural network: each copy turns one of two inputs, say two signatures or two face photos, into a summary, and the two summaries are compared. (1993). The term 'bi-encoder' appears in the 2019 Poly-encoders paper (Samuel Humeau and colleagues, Facebook AI Research); Sentence-BERT (Nils Reimers and Iryna Gurevych, 2019) showed that a fine-tuned BERT bi-encoder cut finding the most similar pair among 10,000 sentences from about 65 hours with a cross-encoder to about 5 seconds.
Usage note
In recommender systems the same design is called a two-tower modeltwo-tower model, n.: A design for recommendation and search in which one neural network turns users or queries into lists of numbers and another does the same for items, so good matches can be found by quick comparison.; retrieval papers often say dual encoder.
Related
- cross-encodercross-encoder, n.: An AI model that reads a question and a candidate passage together, side by side, and outputs one relevance score. Accurate but slow, so it is used to reorder a short list.
- two-tower modeltwo-tower model, n.: A design for recommendation and search in which one neural network turns users or queries into lists of numbers and another does the same for items, so good matches can be found by quick comparison.
- dense retrievaldense retrieval, n.: Finding relevant documents by turning the question and every document into lists of numbers that capture meaning, then picking the documents whose numbers are closest to the question's.
- text embeddingtext embedding, n.: A list of numbers that captures what a piece of text means, so that texts with similar meanings get similar lists. Computers use them to search, sort and group text by meaning.
- embedding modelembedding model, n.: An AI model whose only job is to turn text (or images, or code) into lists of numbers that capture meaning, so a computer can measure how similar two things are.
- Siamese networkSiamese network, n.: A matching machine built from twin copies of one neural network: each copy turns one of two inputs, say two signatures or two face photos, into a summary, and the two summaries are compared.
- contrastive learningcontrastive learning, n.: A way to learn without labels by comparison: the model learns to pull together two versions of the same thing, such as two crops of one photo, and push apart unrelated things.
- cross-encodercross-encoder, n.: An AI model that reads a question and a candidate passage together, side by side, and outputs one relevance score. Accurate but slow, so it is used to reorder a short list.
Sources
- Reimers, N. & Gurevych, I. (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. EMNLP-IJCNLP 2019. arXiv:1908.10084. arxiv.org/abs/1908.10084
- Humeau, S., Shuster, K., Lachaux, M.-A. & Weston, J. (2019). Poly-encoders: Transformer Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring. arXiv:1905.01969. arxiv.org/abs/1905.01969
Mentioned in
cross-encodercross-encoder, n.: An AI model that reads a question and a candidate passage together, side by side, and outputs one relevance score. Accurate but slow, so it is used to reorder a short list., dense retrievaldense retrieval, n.: Finding relevant documents by turning the question and every document into lists of numbers that capture meaning, then picking the documents whose numbers are closest to the question's., embedding modelembedding model, n.: An AI model whose only job is to turn text (or images, or code) into lists of numbers that capture meaning, so a computer can measure how similar two things are., Sentence-BERTSentence-BERT, prop. n.: A way of turning a whole sentence into a list of numbers so that sentences with similar meanings get similar numbers, which makes it fast to search millions of texts by meaning rather than by exact words., two-tower modeltwo-tower model, n.: A design for recommendation and search in which one neural network turns users or queries into lists of numbers and another does the same for items, so good matches can be found by quick comparison., ColBERTColBERT, prop. n.: A search model that compares a question and a document word by word — matching each question word to its best partner in the document — while still letting documents be processed ahead of time for speed., late interactionlate interaction, n.: A search design that keeps a separate number-list for every word of a document and compares them with the question's words at the last moment — more precise than one list per document, yet still fast., Siamese networkSiamese network, n.: A matching machine built from twin copies of one neural network: each copy turns one of two inputs, say two signatures or two face photos, into a summary, and the two summaries are compared.