bag of words
In plain wordsA simple way of turning text into numbers by counting how often each word appears and ignoring word order — like describing a recipe only by its shopping list.
A way of encoding a text that records only which words it contains and how often, as if the words had been cut apart and shaken together in a bag: the result is a vector with one dimension per vocabulary item holding a count, a 0/1 indicator or a weight such as TF-IDFTF-IDF, abbr.: A simple score for picking out the words that characterize a document: a word counts for more the more often it appears there, and for less the more of the collection's other documents also use it., and nothing about sentence structure or the sequence of words survives. It turns documents of any length into fixed-length feature vectors for text classificationtext classification, n.: Sorting pieces of text into categories automatically — what topic an article covers, whether a message is spam, which team should handle a customer's complaint., clustering, topic modelstopic model, n.: A method that reads a large pile of documents and discovers the themes running through them, describing each theme by its typical words and each document by its mix of themes. and information retrievalinformation retrieval, n.: The study and engineering of search: how to organize huge piles of documents so that the few that matter to someone's question can be found and ranked quickly. AI systems rely on it to look things up., and remained a strong baseline for decades because which words occur says much about what a document is about. Its limits — 'dog bites man' equals 'man bites dog', synonyms occupy unrelated dimensions, and vectors are high-dimensional and sparse — motivated n-gram features, word embeddingsword embedding, n.: A list of numbers standing for a word, learned so that words used in similar ways get similar lists. It lets a computer measure that 'cat' is closer to 'dog' than to 'car'. and contextual models. Computer vision borrowed the idea as the 'bag of visual words', which counts occurrences of clustered local image features.
A bag-of-words logistic regression still beats our fine-tuned model on the old spam set.
Origin
The phrase appears in Zellig Harris's 1954 article 'Distributional Structure', in a passage arguing that language is more than a bag of words. As a technique it underlies the vector space modelvector space model, n.: An early, still-influential idea for search: treat every document and query as an arrow in a space with one direction per word, and rank documents by how closely their arrows point the same way as the query's. of retrieval developed by Gerard Salton and colleagues in the 1960s and 1970s.
Related
- TF-IDFTF-IDF, abbr.: A simple score for picking out the words that characterize a document: a word counts for more the more often it appears there, and for less the more of the collection's other documents also use it.
- text classificationtext classification, n.: Sorting pieces of text into categories automatically — what topic an article covers, whether a message is spam, which team should handle a customer's complaint.
- vector space modelvector space model, n.: An early, still-influential idea for search: treat every document and query as an arrow in a space with one direction per word, and rank documents by how closely their arrows point the same way as the query's.
- information retrievalinformation retrieval, n.: The study and engineering of search: how to organize huge piles of documents so that the few that matter to someone's question can be found and ranked quickly. AI systems rely on it to look things up.
- word embeddingword embedding, n.: A list of numbers standing for a word, learned so that words used in similar ways get similar lists. It lets a computer measure that 'cat' is closer to 'dog' than to 'car'.
- topic modeltopic model, n.: A method that reads a large pile of documents and discovers the themes running through them, describing each theme by its typical words and each document by its mix of themes.
- distributional hypothesisdistributional hypothesis, n.: The idea that words found in similar company — surrounded by similar words — tend to be similar in meaning, the insight that lets computers learn something about word meaning from large amounts of text alone.
- word embeddingword embedding, n.: A list of numbers standing for a word, learned so that words used in similar ways get similar lists. It lets a computer measure that 'cat' is closer to 'dog' than to 'car'.
Sources
- Harris, Z. S. (1954). Distributional Structure. Word 10(2–3), 146–162.
Mentioned in
text classificationtext classification, n.: Sorting pieces of text into categories automatically — what topic an article covers, whether a message is spam, which team should handle a customer's complaint., distributional hypothesisdistributional hypothesis, n.: The idea that words found in similar company — surrounded by similar words — tend to be similar in meaning, the insight that lets computers learn something about word meaning from large amounts of text alone., stemmingstemming, n.: Cutting words down to a common root so that variants count as the same word — 'connected', 'connecting' and 'connection' all become 'connect' — to help search and text analysis.