Skip to content
Last Dictionary
Plane
AI Engineering & MLOpsLanguage Models & TransformersDeep Learning

text embedding

nounAlso: sentence embedding, document embedding, passage embedding

In plain wordsA list of numbers that captures what a piece of text means, so that texts with similar meanings get similar lists. Computers use them to search, sort and group text by meaning.

  1. A fixed-length vector of real numbers, typically with a few hundred to a few thousand dimensions, produced by an embedding modelembedding model, n.: An AI model whose only job is to turn text (or images, or code) into lists of numbers that capture meaning, so a computer can measure how similar two things are. to represent a sentence, passage or document so that semantically similar texts map to nearby points, usually compared by cosine similaritycosine similarity, n.: A score from −1 to 1 for how closely two lists of numbers point the same way, ignoring their size. It is the standard test of whether two texts mean similar things. or dot productdot product, n.: A score of how well two lists of numbers line up, found by multiplying matching entries and adding the products. It is large when the lists point the same way and zero when they are at right angles.. Unlike word embeddingsword embedding, n.: A list of numbers standing for a word, learned so that words used in similar ways get similar lists. It lets a computer measure that 'cat' is closer to 'dog' than to 'car'., which give each word one vector, a text embedding summarizes a whole span, usually by pooling the contextual token representations of a transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it. encoder or by taking one designated token's final state. Text embeddings are the representation behind semantic searchsemantic search, n.: Search that understands what you mean, not just the words you typed, so a search for 'cheap places to stay' also finds 'budget hotels'., dense retrievaldense retrieval, n.: Finding relevant documents by turning the question and every document into lists of numbers that capture meaning, then picking the documents whose numbers are closest to the question's. and retrieval-augmented generationretrieval-augmented generation, n.: Giving an AI an open-book exam: before answering, it looks up relevant passages in a set of documents and bases its reply on them, often citing where each point came from., and are also used for clustering, deduplication, classification, recommendation and semantic cachingsemantic caching, n.: Saving an AI system's past answers and reusing one whenever a new question means the same as an old one, even if worded differently, to save time and money.. Embeddings from different models live in different spaces and cannot be compared, so changing models means re-embedding the whole collection.

    We store a text embedding for every support article and look up the nearest ones for each new ticket.

Origin

Precursors include latent semantic analysis (1990) and paragraph vectors (Quoc Le and Tomas Mikolov, 2014); neural sentence encoders such as InferSent (2017), the Universal Sentence Encoder (2018) and Sentence-BERT (2019) made general-purpose text embeddings practical. Hosted embedding APIs from 2022 and the 2023 boom in retrieval-augmented generation turned them into everyday infrastructure.

Usage note

'Embedding' alone also names the layer inside a model that maps tokens to vectors; see embeddingembedding, n.: A list of numbers that stands for a word, picture or other item, arranged so that similar items get similar lists. It lets a computer measure similarity of meaning as distance..

Related

See also
  • embeddingembedding, n.: A list of numbers that stands for a word, picture or other item, arranged so that similar items get similar lists. It lets a computer measure similarity of meaning as distance.
  • embedding modelembedding model, n.: An AI model whose only job is to turn text (or images, or code) into lists of numbers that capture meaning, so a computer can measure how similar two things are.
  • semantic searchsemantic search, n.: Search that understands what you mean, not just the words you typed, so a search for 'cheap places to stay' also finds 'budget hotels'.
  • vector databasevector database, n.: A database built to store embeddings — lists of numbers that represent meaning — and to quickly find the stored items most similar to a new one. It is the memory behind many AI search tools.
  • dense retrievaldense retrieval, n.: Finding relevant documents by turning the question and every document into lists of numbers that capture meaning, then picking the documents whose numbers are closest to the question's.
  • cosine similaritycosine similarity, n.: A score from −1 to 1 for how closely two lists of numbers point the same way, ignoring their size. It is the standard test of whether two texts mean similar things.
  • Matryoshka embeddingMatryoshka embedding, n.: An embedding built like a set of Russian nesting dolls: its first few numbers already give a rough summary, so you can cut it short to save space and still search reasonably well.
  • MTEBMTEB, prop. n.: A large public test suite that scores AI models which turn text into lists of numbers on many different jobs, such as searching, grouping, classifying and matching translations, so people can compare them on one leaderboard.

Sources

  1. Reimers, N. & Gurevych, I. (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. EMNLP-IJCNLP 2019. arXiv:1908.10084. arxiv.org/abs/1908.10084
  2. Muennighoff, N., Tazi, N., Magne, L. & Reimers, N. (2023). MTEB: Massive Text Embedding Benchmark. EACL 2023. arXiv:2210.07316. arxiv.org/abs/2210.07316

Mentioned in

embeddingembedding, n.: A list of numbers that stands for a word, picture or other item, arranged so that similar items get similar lists. It lets a computer measure similarity of meaning as distance., retrieval-augmented generationretrieval-augmented generation, n.: Giving an AI an open-book exam: before answering, it looks up relevant passages in a set of documents and bases its reply on them, often citing where each point came from., approximate nearest neighbor searchapproximate nearest neighbor search, n.: A family of shortcuts for finding the most similar items in a huge collection without checking every one — accepting an occasional miss in exchange for answers thousands of times faster., bi-encoderbi-encoder, n.: An AI design that turns a question and a document into number lists separately, so documents can be processed once in advance and compared with any new question in a flash., chunkingchunking, n.: Cutting long documents into smaller pieces before storing them for AI search, so the system can find and hand the model just the relevant part instead of a whole manual., CohereCohere, prop. n.: A Canadian AI company, founded in Toronto in 2019, that builds language models and search tools for businesses and governments rather than consumer chatbots., cosine similaritycosine similarity, n.: A score from −1 to 1 for how closely two lists of numbers point the same way, ignoring their size. It is the standard test of whether two texts mean similar things., dense retrievaldense retrieval, n.: Finding relevant documents by turning the question and every document into lists of numbers that capture meaning, then picking the documents whose numbers are closest to the question's., embedding modelembedding model, n.: An AI model whose only job is to turn text (or images, or code) into lists of numbers that capture meaning, so a computer can measure how similar two things are., encoder-only transformerencoder-only transformer, n.: A transformer that reads a whole text at once, in both directions, to understand it rather than to write new text. Such models power search, sorting documents by topic and spotting names in text., FAISSFAISS, prop. n.: A free software library from Meta for searching huge collections of embeddings very fast, on ordinary processors or graphics cards. Many AI search systems use it under the hood., GemmaGemma, prop. n.: Google's family of free-to-download AI language models: smaller relatives of its Gemini systems, compact enough to run on a single computer and, in some versions, on a phone., MTEBMTEB, prop. n.: A large public test suite that scores AI models which turn text into lists of numbers on many different jobs, such as searching, grouping, classifying and matching translations, so people can compare them on one leaderboard., semantic searchsemantic search, n.: Search that understands what you mean, not just the words you typed, so a search for 'cheap places to stay' also finds 'budget hotels'., Sentence-BERTSentence-BERT, prop. n.: A way of turning a whole sentence into a list of numbers so that sentences with similar meanings get similar numbers, which makes it fast to search millions of texts by meaning rather than by exact words., vector databasevector database, n.: A database built to store embeddings — lists of numbers that represent meaning — and to quickly find the stored items most similar to a new one. It is the memory behind many AI search tools., vector searchvector search, n.: Searching by closeness instead of exact match: the query and all stored items are lists of numbers, and the search returns the items whose numbers are most similar to the query's., word embeddingword embedding, n.: A list of numbers standing for a word, learned so that words used in similar ways get similar lists. It lets a computer measure that 'cat' is closer to 'dog' than to 'car'., Matryoshka embeddingMatryoshka embedding, n.: An embedding built like a set of Russian nesting dolls: its first few numbers already give a rough summary, so you can cut it short to save space and still search reasonably well., semantic textual similaritysemantic textual similarity, n.: How close two sentences are in meaning, scored from 'about different things' to 'say exactly the same thing' — and the tests that check whether a computer's scores match people's judgments.