The Pile
In plain wordsA large free collection of English text released in 2020 by the volunteer group EleutherAI, mixing 22 sources from books and science papers to code, used to train early open language models.
An open training corpuscorpus, n.: A large collection of written or spoken language, gathered so that people or computers can study it or learn from it — like a library assembled for a language model. for large language modelslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. in which EleutherAI combined 22 deliberately varied parts into 825 GiB of mostly English text: web text (filtered Common CrawlCommon Crawl, prop. n.: A nonprofit's free archive of the public web, begun in 2008 and now refreshed with snapshots of billions of pages every month or two. Most large language models learned much of what they know from text taken from it., called Pile-CC, and OpenWebText2, an open replication of WebTextWebText, prop. n.: The private collection of about eight million web pages, chosen because Reddit users had linked to them, that OpenAI used in 2019 to train GPT-2.), academic writing (PubMed Central, arXiv), books (Books3Books3, prop. n.: A collection of nearly 200,000 pirated e-books, assembled in 2020 and used to train several AI language models, which became central to lawsuits by authors over AI training., Project Gutenberg), code from GitHub, legal and patent text (US court opinions, patents), questions and conversation (Stack Exchange, chat logs, film subtitles) and smaller sources such as English Wikipedia, mathematics problems and the Enron emails. Components were weighted, with sources judged higher in quality repeated for more than one epoch. The Pile trained EleutherAI's GPT-Neo, GPT-J, GPT-NeoX-20B and Pythia models and was used by other developers, and its fully published composition made it a reference point for research on pretraining datapretraining data, n.: The enormous body of text, code and other material — mostly from the web — that a large AI model first learns from, before it is tuned to follow instructions or chat..
Pythia's checkpoints are useful precisely because every model saw the Pile in the same order.
Origin
Released on 31 December 2020 by EleutherAI, then a volunteer research collective, and described in 'The Pile: An 800GB Dataset of Diverse Text for Language Modeling' by Leo Gao, Stella Biderman, Sid Black and colleagues. A successor, the Common Pile (2025), is restricted to public-domain and openly licensed text.
Usage note
Its Books3 component, taken from a pirate e-book site, made The Pile a focus of copyright suits; after Books3 was removed through takedown notices in 2023, The Pile itself was no longer available from its original host by 2024, though copies remained online. The title says 800GB; the abstract gives 825 GiB (about 886 GB).
Related
- Books3Books3, prop. n.: A collection of nearly 200,000 pirated e-books, assembled in 2020 and used to train several AI language models, which became central to lawsuits by authors over AI training.
- Common CrawlCommon Crawl, prop. n.: A nonprofit's free archive of the public web, begun in 2008 and now refreshed with snapshots of billions of pages every month or two. Most large language models learned much of what they know from text taken from it.
- pretraining datapretraining data, n.: The enormous body of text, code and other material — mostly from the web — that a large AI model first learns from, before it is tuned to follow instructions or chat.
- corpuscorpus, n.: A large collection of written or spoken language, gathered so that people or computers can study it or learn from it — like a library assembled for a language model.
- C4C4, prop. n.: A large collection of English web text, cleaned up from a 2019 web archive snapshot by Google researchers, that became a standard ingredient for training language models.
- data mixturedata mixture, n.: The recipe of different kinds of data a model is trained on — how much web text, code, books, math, science and other languages — which strongly shapes what the finished model is good at.
Sources
- Gao, L. et al. (2020). The Pile: An 800GB Dataset of Diverse Text for Language Modeling. arXiv:2101.00027. arxiv.org/abs/2101.00027v1
Mentioned in
corpuscorpus, n.: A large collection of written or spoken language, gathered so that people or computers can study it or learn from it — like a library assembled for a language model., data mixturedata mixture, n.: The recipe of different kinds of data a model is trained on — how much web text, code, books, math, science and other languages — which strongly shapes what the finished model is good at., EleutherAIEleutherAI, prop. n.: An open-science research group, begun by online volunteers in 2020, that released some of the first large language models anyone could download, along with the text collections used to train them., BookCorpusBookCorpus, prop. n.: A collection of about 11,000 free self-published novels gathered from the web in 2015, used to train the first GPT model and BERT despite problems with consent and duplicates found later., Books3Books3, prop. n.: A collection of nearly 200,000 pirated e-books, assembled in 2020 and used to train several AI language models, which became central to lawsuits by authors over AI training., WebTextWebText, prop. n.: The private collection of about eight million web pages, chosen because Reddit users had linked to them, that OpenAI used in 2019 to train GPT-2.