embedding
In plain wordsA list of numbers that stands for a word, picture or other item, arranged so that similar items get similar lists. It lets a computer measure similarity of meaning as distance.
- 1
Machine learning
A dense vector of real numbers that a model assigns to an item — a tokentoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens., document, image, product, user or graph node — or the learned function that assigns such vectors. The vectors live in a space of fixed dimension, typically hundreds to a few thousand, far smaller than the number of distinct items or the size of the raw input, and training arranges that space so distance carries meaning: related items end up close together, as measured by cosine similaritycosine similarity, n.: A score from −1 to 1 for how closely two lists of numbers point the same way, ignoring their size. It is the standard test of whether two texts mean similar things. or dot productdot product, n.: A score of how well two lists of numbers line up, found by multiplying matching entries and adding the products. It is large when the lists point the same way and zero when they are at right angles., and some directions come to stand for attributes. The arrangement emerges from whatever objective the model is trained on, such as predicting neighboring words (word2vecword2vec, prop. n.: A 2013 Google technique that learns a list of numbers for each word by training a very small network to predict which words appear near it across billions of words of text.), matching paired inputs (contrastive learningcontrastive learning, n.: A way to learn without labels by comparison: the model learns to pull together two versions of the same thing, such as two crops of one photo, and push apart unrelated things., CLIPCLIP, prop. n.: An OpenAI model that learned to match pictures with their captions from hundreds of millions of web examples, so it can judge how well any piece of text describes any image.) or predicting the next token. Because related items receive nearby vectors, what a model learns about one carries over to the others, which one-hot codesone-hot encoding, n.: Turning a category such as 'red', 'green' or 'blue' into a row of zeros with a single one marking which category it is, so that a model can use it as numbers., with every item equally far from every other, do not allow. Dedicated embedding modelsembedding model, n.: An AI model whose only job is to turn text (or images, or code) into lists of numbers that capture meaning, so a computer can measure how similar two things are. produce text embeddingstext embedding, n.: A list of numbers that captures what a piece of text means, so that texts with similar meanings get similar lists. Computers use them to search, sort and group text by meaning. for semantic searchsemantic search, n.: Search that understands what you mean, not just the words you typed, so a search for 'cheap places to stay' also finds 'budget hotels'., retrievalretrieval-augmented generation, n.: Giving an AI an open-book exam: before answering, it looks up relevant passages in a set of documents and bases its reply on them, often citing where each point came from., recommendationrecommender system, n.: Software that guesses which songs, products, posts or jobs you are most likely to want and shows those first, as when a video app lines up the next clip or a shop suggests an add-on., clustering and deduplication, usually stored in a vector databasevector database, n.: A database built to store embeddings — lists of numbers that represent meaning — and to quickly find the stored items most similar to a new one. It is the memory behind many AI search tools..
We embed every support ticket and retrieve the five nearest neighbors for each new question.
- 2
Network layer
A trainable lookup table, a matrix with one row per vocabulary item, that converts integer IDs into vectors at the input of a network (an 'embedding layer'); it is equivalent to multiplying a one-hot vector by a weight matrix. In a large language modellarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. the token-embedding matrix is sometimes tied to the output unembeddingunembedding, n.: The last step of a language model, which turns its internal state into a score for every token in its vocabulary; the next token is then chosen from those scores. (weight sharingweight sharing, n.: Reusing the same learned numbers in many places in a network, for example one edge detector applied at every spot in an image, so the network needs fewer parameters and learns from less data.), and position information is added or applied separately (positional encodingpositional encoding, n.: Extra information added to each word so a transformer knows where it sits in the text. Without it, 'dog bites man' and 'man bites dog' would look the same to the model.).
- 3
Mathematics
An injective, structure-preserving map of one mathematical object into another, such as a graph or manifold into Euclidean space. This is the source of the machine-learning usage and survives in nonlinear dimensionality reductiondimensionality reduction, n.: Squeezing data that has many measurements per item into far fewer numbers while keeping the patterns that matter, so it is easier to visualize, store or learn from., where methods such as locally linear embedding and t-SNEt-SNE, n.: A technique for drawing data that have many measurements per item as a flat map in which similar items land close together, widely used to look for groups in AI models' internal data or in genetic data. 'embed' data in two or three dimensions.
Origin
Borrowed from mathematics. Learned vectors for symbols go back to Geoffrey Hinton's 1986 work on 'distributed representations', and Yoshua Bengio and colleagues' neural language model (NIPS 2000; JMLR 2003) learned a feature vector for every word jointly with the model. When 'embedding' was first used for such learned vectors is not established, but 'word embedding' was common by about 2010, following work such as Ronan Collobert and Jason Weston's (2008). Tomas Mikolov and colleagues' word2vec (2013) made embeddings ubiquitous.
Usage note
Both noun and verb ('we embed each document'). Embedding similarity reflects the training objective, not meaning in any guaranteed sense; two models' embeddings are not interchangeable.
Related
- word embeddingword embedding, n.: A list of numbers standing for a word, learned so that words used in similar ways get similar lists. It lets a computer measure that 'cat' is closer to 'dog' than to 'car'.
- text embeddingtext embedding, n.: A list of numbers that captures what a piece of text means, so that texts with similar meanings get similar lists. Computers use them to search, sort and group text by meaning.
- embedding modelembedding model, n.: An AI model whose only job is to turn text (or images, or code) into lists of numbers that capture meaning, so a computer can measure how similar two things are.
- vector databasevector database, n.: A database built to store embeddings — lists of numbers that represent meaning — and to quickly find the stored items most similar to a new one. It is the memory behind many AI search tools.
- latent spacelatent space, n.: The hidden internal map a model builds of its data, where each item becomes a point and similar items sit near each other. Moving smoothly through it gradually changes what the model represents or generates.
- representation learningrepresentation learning, n.: Letting a computer work out for itself how best to describe its data, such as the edges and shapes in images or the meanings of words, instead of relying on descriptions people design by hand.
- cosine similaritycosine similarity, n.: A score from −1 to 1 for how closely two lists of numbers point the same way, ignoring their size. It is the standard test of whether two texts mean similar things.
- unembeddingunembedding, n.: The last step of a language model, which turns its internal state into a score for every token in its vocabulary; the next token is then chosen from those scores.
Sources
- Bengio, Y., Ducharme, R., Vincent, P. & Jauvin, C. (2003). A Neural Probabilistic Language Model. Journal of Machine Learning Research, 3, 1137–1155.
- Mikolov, T., Chen, K., Corrado, G. & Dean, J. (2013). Efficient Estimation of Word Representations in Vector Space. ICLR 2013 Workshop.
Mentioned in
AI alignmentAI alignment, n.: Making an AI system genuinely try to do what the people it serves want — not a literal-minded, distorted or self-serving version of it. Also the research field working on this., facial recognitionfacial recognition, n.: Technology that identifies or checks who someone is by comparing a picture of their face with stored photos, used by phones, passport gates and police databases., layerlayer, n.: One stage in a neural network: a group of artificial neurons that all take the same input, transform it, and pass the result on to the next stage., transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it., vectorvector, n.: An ordered list of numbers, like map coordinates. AI systems turn words, images and users into long lists of numbers so that similar things end up as nearby points., activation vectoractivation vector, n.: The list of numbers a neural network produces at one layer for one input — a snapshot of its internal state. In steering, also a direction added to that state to shift behavior., biometric databiometric data, n.: Information about your body or behavior — your face, fingerprints, voice, eyes or the way you walk — measured by technology and used to recognize who you are or to guess things about you., click-through rate predictionclick-through rate prediction, n.: Estimating how likely a particular person is to click on a particular ad, search result or recommended item: a prediction made billions of times a day to decide what each user is shown., CLIPCLIP, prop. n.: An OpenAI model that learned to match pictures with their captions from hundreds of millions of web examples, so it can judge how well any piece of text describes any image., clusteringclustering, n.: Automatically sorting things into groups of similar items when nobody has said in advance what the groups should be, like sorting a drawer of odd socks into piles by color and size., cold start problemcold start problem, n.: The trouble a recommendation system has with brand-new users or items: with no history to learn from, it has little basis for good suggestions., content-based filteringcontent-based filtering, n.: Recommending items that resemble what you already liked, judged by the items' own features, such as a film's genre, cast and plot, rather than by other people's choices., contrastive learningcontrastive learning, n.: A way to learn without labels by comparison: the model learns to pull together two versions of the same thing, such as two crops of one photo, and push apart unrelated things., cosine similaritycosine similarity, n.: A score from −1 to 1 for how closely two lists of numbers point the same way, ignoring their size. It is the standard test of whether two texts mean similar things., data driftdata drift, n.: When the information flowing into an AI model in real use gradually stops resembling the data it was trained on — new customers, new slang, new cameras — making its predictions less reliable., deduplicationdeduplication, n.: Removing repeated copies of the same or nearly the same content from a dataset, so a model does not over-learn whatever happens to be copied many times online., dimensionalitydimensionality, n.: How many separate numbers are needed to describe something. A point on a map needs two; an AI's description of a single word may use thousands., dimensionality reductiondimensionality reduction, n.: Squeezing data that has many measurements per item into far fewer numbers while keeping the patterns that matter, so it is easier to visualize, store or learn from., featurefeature, n.: One piece of information about each example that a model uses, like a house's size or a pixel's color, or a pattern a model has learned to detect inside itself., feature extractionfeature extraction, n.: Turning raw data such as pixels or sound into a short list of telling numbers — computed by hand-designed rules or by an AI system trained earlier — that a new model can learn from or compare., feature scalingfeature scaling, n.: Putting input measurements on comparable scales, so that income in dollars and age in years, say, both span similar ranges and no feature dominates just because of its units., k-nearest neighborsk-nearest neighbors, n.: Predicting something about a new example by finding the few most similar examples already known and going with their majority answer, or their average., LangChainLangChain, prop. n.: A popular free software toolkit, and the company behind it, for building applications that connect AI language models to documents, databases, tools and multi-step workflows., latent spacelatent space, n.: The hidden internal map a model builds of its data, where each item becomes a point and similar items sit near each other. Moving smoothly through it gradually changes what the model represents or generates.