TF-IDF
In plain wordsA simple score for picking out the words that characterize a document: a word counts for more the more often it appears there, and for less the more of the collection's other documents also use it.
A term-weighting scheme that multiplies a term's frequency in a document (tf), which rewards words the document uses often, by its inverse document frequency (idf), the logarithm of the collection size divided by the number of documents containing the term, which discounts words that appear everywhere. The weights turn each document into a sparse vector used for keyword search in the vector space modelvector space model, n.: An early, still-influential idea for search: treat every document and query as an arrow in a space with one direction per word, and rank documents by how closely their arrows point the same way as the query's. (usually scored by cosine similaritycosine similarity, n.: A score from −1 to 1 for how closely two lists of numbers point the same way, ignoring their size. It is the standard test of whether two texts mean similar things.), as features for text classificationtext classification, n.: Sorting pieces of text into categories automatically — what topic an article covers, whether a message is spam, which team should handle a customer's complaint. and clusteringclustering, n.: Automatically sorting things into groups of similar items when nobody has said in advance what the groups should be, like sorting a drawer of odd socks into piles by color and size., and for keyword extraction. Many variants exist — logarithmic or binary tf, smoothed idf, length normalization — and BM25BM25, prop. n.: A decades-old formula that search engines use to score how well a document matches a query, rewarding rare query words and repeated matches — with diminishing returns — and correcting for long documents. can be read as a refined tf-idf whose term-frequency credit saturates.
A TF-IDF baseline with logistic regression still gets 88% on the ticket-routing task.
Origin
Inverse document frequency was proposed by Karen Spärck Jones in 1972 ('A statistical interpretation of term specificity and its application in retrieval', Journal of Documentation). Weighting by within-document frequency goes back to Hans Peter Luhn's work at IBM in the late 1950s. Gerard Salton's SMART group combined and systematized tf·idf weighting in the 1970s and 1980s.
Usage note
In the formula, N is the number of documents in the collection and df(t) the number that contain term t.
Related
- BM25BM25, prop. n.: A decades-old formula that search engines use to score how well a document matches a query, rewarding rare query words and repeated matches — with diminishing returns — and correcting for long documents.
- vector space modelvector space model, n.: An early, still-influential idea for search: treat every document and query as an arrow in a space with one direction per word, and rank documents by how closely their arrows point the same way as the query's.
- sparse retrievalsparse retrieval, n.: Finding documents by the words they contain, giving more weight to rare, telling words — the classic way search engines work.
- inverted indexinverted index, n.: A lookup table that lists, for every word, all the documents containing it — like the index at the back of a book — so a search engine can find matches almost instantly.
- information retrievalinformation retrieval, n.: The study and engineering of search: how to organize huge piles of documents so that the few that matter to someone's question can be found and ranked quickly. AI systems rely on it to look things up.
- text classificationtext classification, n.: Sorting pieces of text into categories automatically — what topic an article covers, whether a message is spam, which team should handle a customer's complaint.
Sources
- Spärck Jones, K. (1972). A statistical interpretation of term specificity and its application in retrieval. Journal of Documentation 28(1): 11–21.
- Salton, G. & Buckley, C. (1988). Term-weighting approaches in automatic text retrieval. Information Processing & Management 24(5): 513–523.
Mentioned in
bag of wordsbag of words, n.: A simple way of turning text into numbers by counting how often each word appears and ignoring word order — like describing a recipe only by its shopping list., BM25BM25, prop. n.: A decades-old formula that search engines use to score how well a document matches a query, rewarding rare query words and repeated matches — with diminishing returns — and correcting for long documents., inverted indexinverted index, n.: A lookup table that lists, for every word, all the documents containing it — like the index at the back of a book — so a search engine can find matches almost instantly., sparse retrievalsparse retrieval, n.: Finding documents by the words they contain, giving more weight to rare, telling words — the classic way search engines work., text classificationtext classification, n.: Sorting pieces of text into categories automatically — what topic an article covers, whether a message is spam, which team should handle a customer's complaint., stemmingstemming, n.: Cutting words down to a common root so that variants count as the same word — 'connected', 'connecting' and 'connection' all become 'connect' — to help search and text analysis., vector space modelvector space model, n.: An early, still-influential idea for search: treat every document and query as an arrow in a space with one direction per word, and rank documents by how closely their arrows point the same way as the query's.