word2vec
In plain wordsA 2013 Google technique that learns a list of numbers for each word by training a very small network to predict which words appear near it across billions of words of text.
A method, released with an open-source tool, for learning word embeddingsword embedding, n.: A list of numbers standing for a word, learned so that words used in similar ways get similar lists. It lets a computer measure that 'cat' is closer to 'dog' than to 'car'. from unannotated text by training a very small neural network — in effect one linear projection feeding an output classifier — on a word-prediction task. It comes in two variants: continuous bag-of-words (CBOW) guesses a word from the average of the vectors of the words around it, and skip-gram uses a word to guess each of its neighbors within a small window. Training is made cheap by replacing the full softmaxsoftmax, n.: A formula that turns a list of raw scores into probabilities — all positive and adding up to one — with the biggest share going to the highest score. over the vocabulary with hierarchical softmax or negative sampling — a logistic-regression task of distinguishing observed word–context pairs from randomly drawn 'negative' pairs — and by subsampling very frequent words. Vectors could be learned from a 1.6-billion-word dataset in under a day, and they displayed linear analogies such as king − man + woman ≈ queen. Omer Levy and Yoav Goldberg (2014) showed that skip-gram with negative sampling implicitly factorizes a shifted pointwise mutual information matrix, linking it to count-based methods such as GloVeGloVe, prop. n.: A 2014 Stanford method for learning number lists that represent words, built from counts of how often each pair of words appears near each other across a very large body of text..
We trained skip-gram vectors on the product catalog to find similar items.
Origin
Introduced by Tomas Mikolov, Kai Chen, Greg Corrado and Jeffrey Dean at Google in January 2013; a second paper with Ilya Sutskever (NIPS 2013) added negative sampling, subsampling and phrase vectors, and Google released the C code that year. Its speed and analogy demonstrations made pretrained embeddings ubiquitous in NLP.
Usage note
The NIPS 2013 paper received the NeurIPS Test of Time Award in 2023. The idea of training on word–context prediction has since been applied to graphs, products and genes ('anything2vec').
Related
- word embeddingword embedding, n.: A list of numbers standing for a word, learned so that words used in similar ways get similar lists. It lets a computer measure that 'cat' is closer to 'dog' than to 'car'.
- GloVeGloVe, prop. n.: A 2014 Stanford method for learning number lists that represent words, built from counts of how often each pair of words appears near each other across a very large body of text.
- embeddingembedding, n.: A list of numbers that stands for a word, picture or other item, arranged so that similar items get similar lists. It lets a computer measure similarity of meaning as distance.
- language modellanguage model, n.: A system that has learned how likely different sequences of words are, so it can predict the next word in a text and, by doing that repeatedly, write text of its own.
- representation learningrepresentation learning, n.: Letting a computer work out for itself how best to describe its data, such as the edges and shapes in images or the meanings of words, instead of relying on descriptions people design by hand.
Sources
- Mikolov, T., Chen, K., Corrado, G. & Dean, J. (2013). Efficient Estimation of Word Representations in Vector Space. arXiv:1301.3781. arxiv.org/abs/1301.3781
- Mikolov, T., Sutskever, I., Chen, K., Corrado, G. & Dean, J. (2013). Distributed Representations of Words and Phrases and their Compositionality. NIPS 2013. arxiv.org/abs/1310.4546
Mentioned in
embeddingembedding, n.: A list of numbers that stands for a word, picture or other item, arranged so that similar items get similar lists. It lets a computer measure similarity of meaning as distance., deep learning revolutiondeep learning revolution, n.: The rapid takeover of AI by many-layered neural networks from about 2012, when they suddenly began beating older methods at recognizing images and speech, leading to today's AI systems., ICLRICLR, prop. n.: A major deep-learning conference, held every spring since 2013, where the reviews of each submitted paper are posted online for anyone to read., Jeff DeanJeff Dean, prop. n.: An American computer scientist who built much of Google's computing infrastructure, co-founded the Google Brain deep-learning team and was Google's chief scientist before leaving in 2026 to co-found a startup., word embeddingword embedding, n.: A list of numbers standing for a word, learned so that words used in similar ways get similar lists. It lets a computer measure that 'cat' is closer to 'dog' than to 'car'., distributional hypothesisdistributional hypothesis, n.: The idea that words found in similar company — surrounded by similar words — tend to be similar in meaning, the insight that lets computers learn something about word meaning from large amounts of text alone., GloVeGloVe, prop. n.: A 2014 Stanford method for learning number lists that represent words, built from counts of how often each pair of words appears near each other across a very large body of text., linear representation hypothesislinear representation hypothesis, n.: The idea that a neural network stores many concepts as simple directions inside its internal activity, so how much of a concept is present can be read off by measuring along its direction., mutual informationmutual information, n.: How much knowing one thing tells you about another — for example, how much a word reveals about the topic of a document. It is zero when the two are unrelated., task arithmetictask arithmetic, n.: Editing an AI model by adding or subtracting the changes that fine-tuning made to it, as if skills were arrows that can be added together or reversed., Tomas MikolovTomas Mikolov, prop. n.: A Czech computer scientist who created word2vec, a fast way to turn words into lists of numbers that capture their meaning, and pioneered language models built from recurrent networks., wav2vecwav2vec, prop. n.: A family of Meta speech models that learn the sounds of language from thousands of hours of unlabeled recordings, so that only minutes of transcribed speech are needed to build a recognizer.