deduplication
In plain wordsRemoving repeated copies of the same or nearly the same content from a dataset, so a model does not over-learn whatever happens to be copied many times online.
The removal of duplicate and near-duplicate content from a dataset. Exact methods hash whole documents, lines or fixed-length substrings (suffix arrays can find every repeated span above a length threshold); near-duplicate methods compare sets of word or character n-grams using MinHashMinHash, n.: A shortcut for spotting near-copies among millions of documents: each document is boiled down to a short fingerprint, and similar documents end up with similar fingerprints. with locality-sensitive hashinglocality-sensitive hashing, n.: A trick for finding look-alikes in a huge collection without comparing every pair: items get short codes designed so that similar items tend to share a code, and only items sharing one are compared., or SimHash fingerprints, catching pages that differ only in dates, ads or boilerplate; semantic methods cluster embeddingsembedding, n.: A list of numbers that stands for a word, picture or other item, arranged so that similar items get similar lists. It lets a computer measure similarity of meaning as distance. to drop documents that say the same thing in different words. Web crawls are highly repetitive — templates, mirrors, spam, licenses, quoted text — and duplicated text is overweighted in training, which increases memorizationmemorization, n.: When an AI model stores specific pieces of its training data — a paragraph, a phone number, an image — closely enough to reproduce them, instead of only learning general patterns. and regurgitation, wastes compute and inflates benchmark scores when test items recur in training data. Deduplicating C4 removed one 61-word sentence that appeared more than 60,000 times, and models trained on deduplicated data emitted memorized text about ten times less often.
Near-dedup at a Jaccard threshold of 0.8 removed 38% of the crawl.
Origin
Near-duplicate detection was developed for web search: Andrei Broder's shingling and MinHash at AltaVista (1997) and Moses Charikar's SimHash (2002). Katherine Lee and colleagues' 'Deduplicating Training Data Makes Language Models Better' (2021) made thorough deduplication a standard stage of language-model data pipelines.
Usage note
How much to deduplicate is a trade-off: FineWeb's builders found that deduplicating each crawl separately gave better models than deduplicating across all crawls, and some repetition of high-quality data helps. Removing copies of test sets is called decontamination (see data contaminationdata contamination, n.: When the test questions used to grade an AI system were already in its training data, so a high score may reflect memorized answers rather than real ability.).
Related
- MinHashMinHash, n.: A shortcut for spotting near-copies among millions of documents: each document is boiled down to a short fingerprint, and similar documents end up with similar fingerprints.
- locality-sensitive hashinglocality-sensitive hashing, n.: A trick for finding look-alikes in a huge collection without comparing every pair: items get short codes designed so that similar items tend to share a code, and only items sharing one are compared.
- Jaccard similarityJaccard similarity, n.: A simple score for how much two collections overlap: the number of items they share divided by the number of distinct items in either. Identical sets score 1; sets with nothing in common score 0.
- memorizationmemorization, n.: When an AI model stores specific pieces of its training data — a paragraph, a phone number, an image — closely enough to reproduce them, instead of only learning general patterns.
- data contaminationdata contamination, n.: When the test questions used to grade an AI system were already in its training data, so a high score may reflect memorized answers rather than real ability.
- data pipelinedata pipeline, n.: The chain of automated steps that collects raw data, cleans and transforms it, and delivers it in a usable form — the factory line that turns messy inputs into training material.
- quality filteringquality filtering, n.: Automatically sorting through huge piles of collected text and keeping only the parts judged useful for training, such as well-written articles, while discarding spam, gibberish and junk.
Sources
- Lee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison-Burch, C. & Carlini, N. (2022). Deduplicating Training Data Makes Language Models Better. ACL 2022. arXiv:2107.06499. arxiv.org/abs/2107.06499
Mentioned in
data cleaningdata cleaning, n.: Finding and fixing errors in a dataset — typos, duplicates, missing values, impossible numbers, wrong labels — before using it, because flawed data produces flawed results., data contaminationdata contamination, n.: When the test questions used to grade an AI system were already in its training data, so a high score may reflect memorized answers rather than real ability., data mixturedata mixture, n.: The recipe of different kinds of data a model is trained on — how much web text, code, books, math, science and other languages — which strongly shapes what the finished model is good at., data pipelinedata pipeline, n.: The chain of automated steps that collects raw data, cleans and transforms it, and delivers it in a usable form — the factory line that turns messy inputs into training material., data qualitydata quality, n.: How well a set of data does its job: whether it is accurate, complete, up to date, consistent and representative — and, for AI training, informative enough to teach useful skills., data-centric AIdata-centric AI, n.: An approach to building AI that focuses on improving the data — fixing labels, adding missing cases, cleaning inconsistencies — rather than endlessly tweaking the model., pretraining datapretraining data, n.: The enormous body of text, code and other material — mostly from the web — that a large AI model first learns from, before it is tuned to follow instructions or chat., canary stringcanary string, n.: A unique marker planted in data to trace what an AI model has absorbed: a code in test files so they can be kept out of training, or a fake secret hidden in training data to measure memorization., FineWebFineWeb, prop. n.: A huge, free collection of cleaned-up English web text — around 15 trillion word fragments — released by Hugging Face in 2024 with a detailed public account of how it was built., Jaccard similarityJaccard similarity, n.: A simple score for how much two collections overlap: the number of items they share divided by the number of distinct items in either. Identical sets score 1; sets with nothing in common score 0., locality-sensitive hashinglocality-sensitive hashing, n.: A trick for finding look-alikes in a huge collection without comparing every pair: items get short codes designed so that similar items tend to share a code, and only items sharing one are compared., MinHashMinHash, n.: A shortcut for spotting near-copies among millions of documents: each document is boiled down to a short fingerprint, and similar documents end up with similar fingerprints., quality filteringquality filtering, n.: Automatically sorting through huge piles of collected text and keeping only the parts judged useful for training, such as well-written articles, while discarding spam, gibberish and junk., The StackThe Stack, prop. n.: A large free collection of computer source code from public repositories, selected mainly for permissive licenses and released from 2022 onward by the BigCode project for training coding models.