Common Crawl
In plain wordsA nonprofit's free archive of the public web, begun in 2008 and now refreshed with snapshots of billions of pages every month or two. Most large language models learned much of what they know from text taken from it.
A public archive of web snapshots that the Common Crawl Foundation, a US nonprofit, collects and gives away for anyone to use. Its crawler, CCBot, gathers a new snapshot every month or two — typically a few billion pages, each crawl labeled like CC-MAIN-2024-10 — and releases raw page captures (WARC files), metadata (WAT) and extracted plain text (WET) free of charge through cloud storage; the cumulative archive is measured in petabytes. Common Crawl is the main raw source of pretraining datapretraining data, n.: The enormous body of text, code and other material — mostly from the web — that a large AI model first learns from, before it is tuned to follow instructions or chat. for open and closed models alike: filtered versions supplied about 60% of the training mix of GPT-3GPT-3, prop. n.: A huge 2020 OpenAI language model that could perform new tasks simply from instructions or a few examples written into its prompt, without retraining. It made prompting a new way of programming. and the bulk of corpora such as C4C4, prop. n.: A large collection of English web text, cleaned up from a 2019 web archive snapshot by Google researchers, that became a standard ingredient for training language models., RefinedWeb, FineWebFineWeb, prop. n.: A huge, free collection of cleaned-up English web text — around 15 trillion word fragments — released by Hugging Face in 2024 with a detailed public account of how it was built. and DCLM. Because it is unfiltered, its text must be extracted, language-identified, deduplicated and quality-filteredquality filtering, n.: Automatically sorting through huge piles of collected text and keeping only the parts judged useful for training, such as well-written articles, while discarding spam, gibberish and junk. before use, and audits have found hate speech, personal data and copyrighted and paywalled material in it.
We started from the 2024 Common Crawl snapshots and kept about 10% of the extracted text.
Origin
Founded in 2007 by the entrepreneur Gil Elbaz to give researchers and startups web-scale data of the kind then held only by large search companies; crawl data has been collected since 2008. Its use as AI training material took off at the end of the 2010s.
Usage note
Common Crawl says it honors robots.txt and removal requests, but a 2025 investigation in The Atlantic reported that paywalled articles and content from publishers who had asked for removal remained in its archives. By 2025 CCBot was among the crawlers most often blocked by major websites.
Related
- web crawlerweb crawler, n.: A program that automatically travels the web, following links from page to page and downloading what it finds — used by search engines, archives and AI companies to gather data.
- C4C4, prop. n.: A large collection of English web text, cleaned up from a 2019 web archive snapshot by Google researchers, that became a standard ingredient for training language models.
- FineWebFineWeb, prop. n.: A huge, free collection of cleaned-up English web text — around 15 trillion word fragments — released by Hugging Face in 2024 with a detailed public account of how it was built.
- pretraining datapretraining data, n.: The enormous body of text, code and other material — mostly from the web — that a large AI model first learns from, before it is tuned to follow instructions or chat.
- robots.txtrobots.txt, n.: A small public file in which a website posts its rules for automated visitors, such as search-engine and AI crawlers, listing areas they are asked to stay out of. Obeying it is voluntary.
- quality filteringquality filtering, n.: Automatically sorting through huge piles of collected text and keeping only the parts judged useful for training, such as well-written articles, while discarding spam, gibberish and junk.
- web scrapingweb scraping, n.: Using software to automatically copy information from websites — text, prices, images — in bulk, which is how much of the data used to train AI was gathered.
Sources
- Common Crawl Foundation. Overview. commoncrawl.org/overview
Mentioned in
datasetdataset, n.: An organized collection of examples gathered for a purpose — such as photos with their captions, or questions with answers — used to teach or test an AI system., corpuscorpus, n.: A large collection of written or spoken language, gathered so that people or computers can study it or learn from it — like a library assembled for a language model., data mixturedata mixture, n.: The recipe of different kinds of data a model is trained on — how much web text, code, books, math, science and other languages — which strongly shapes what the finished model is good at., LAIONLAION, prop. n.: A mostly volunteer group, registered as a German nonprofit, that compiled free lists of billions of online pictures and their captions. Stable Diffusion and other image generators were trained on them., pretraining datapretraining data, n.: The enormous body of text, code and other material — mostly from the web — that a large AI model first learns from, before it is tuned to follow instructions or chat., robots.txtrobots.txt, n.: A small public file in which a website posts its rules for automated visitors, such as search-engine and AI crawlers, listing areas they are asked to stay out of. Obeying it is voluntary., web crawlerweb crawler, n.: A program that automatically travels the web, following links from page to page and downloading what it finds — used by search engines, archives and AI companies to gather data., web scrapingweb scraping, n.: Using software to automatically copy information from websites — text, prices, images — in bulk, which is how much of the data used to train AI was gathered., C4C4, prop. n.: A large collection of English web text, cleaned up from a 2019 web archive snapshot by Google researchers, that became a standard ingredient for training language models., FineWebFineWeb, prop. n.: A huge, free collection of cleaned-up English web text — around 15 trillion word fragments — released by Hugging Face in 2024 with a detailed public account of how it was built., LAION-5BLAION-5B, prop. n.: A free list of nearly six billion links to online images paired with their captions, released in 2022 and used to train image generators such as Stable Diffusion., The PileThe Pile, prop. n.: A large free collection of English text released in 2020 by the volunteer group EleutherAI, mixing 22 sources from books and science papers to code, used to train early open language models.