GloVe
In plain wordsA 2014 Stanford method for learning number lists that represent words, built from counts of how often each pair of words appears near each other across a very large body of text.
A model for learning word embeddingsword embedding, n.: A list of numbers standing for a word, learned so that words used in similar ways get similar lists. It lets a computer measure that 'cat' is closer to 'dog' than to 'car'. from corpus-wide co-occurrence counts rather than by sliding through the text predicting words from their neighbors. It first counts how often each word occurs within a window of every other word across a corpus, then learns word and context vectors, plus biases, whose dot products approximate the logarithms of those counts, minimizing a weighted least-squares loss that down-weights rare co-occurrences and caps the influence of very frequent ones. The design rests on the observation that ratios of co-occurrence probabilities carry meaning. Its authors presented it as uniting two traditions: count-based methods that factorize corpus-wide statistics, such as latent semantic analysislatent semantic analysis, n.: An early technique for capturing word meaning by counting which words appear in which documents, then compressing those counts so that words used in similar contexts end up close together., and prediction-based methods trained on local windows, such as word2vecword2vec, prop. n.: A 2013 Google technique that learns a list of numbers for each word by training a very small network to predict which words appear near it across billions of words of text.. Pretrained GloVe vectors, trained on Wikipedia, newswire and web text, were standard inputs to NLP models until contextual representations displaced them.
For the baseline we used the 100-dimensional GloVe vectors.
Origin
Introduced by Jeffrey Pennington, Richard Socher and Christopher D. Manning of Stanford University at EMNLP 2014 in Doha, Qatar, with pretrained vectors released by the Stanford NLP Group.
Related
- word embeddingword embedding, n.: A list of numbers standing for a word, learned so that words used in similar ways get similar lists. It lets a computer measure that 'cat' is closer to 'dog' than to 'car'.
- word2vecword2vec, prop. n.: A 2013 Google technique that learns a list of numbers for each word by training a very small network to predict which words appear near it across billions of words of text.
- embeddingembedding, n.: A list of numbers that stands for a word, picture or other item, arranged so that similar items get similar lists. It lets a computer measure similarity of meaning as distance.
- latent semantic analysislatent semantic analysis, n.: An early technique for capturing word meaning by counting which words appear in which documents, then compressing those counts so that words used in similar contexts end up close together.
- cosine similaritycosine similarity, n.: A score from −1 to 1 for how closely two lists of numbers point the same way, ignoring their size. It is the standard test of whether two texts mean similar things.
Sources
- Pennington, J., Socher, R. & Manning, C. D. (2014). GloVe: Global Vectors for Word Representation. EMNLP 2014, 1532–1543. aclanthology.org/D14-1162
Mentioned in
word embeddingword embedding, n.: A list of numbers standing for a word, learned so that words used in similar ways get similar lists. It lets a computer measure that 'cat' is closer to 'dog' than to 'car'., word2vecword2vec, prop. n.: A 2013 Google technique that learns a list of numbers for each word by training a very small network to predict which words appear near it across billions of words of text., distributional hypothesisdistributional hypothesis, n.: The idea that words found in similar company — surrounded by similar words — tend to be similar in meaning, the insight that lets computers learn something about word meaning from large amounts of text alone.