WebText
In plain wordsThe private collection of about eight million web pages, chosen because Reddit users had linked to them, that OpenAI used in 2019 to train GPT-2.
A text corpus of about 40 GB from roughly 8 million documents, built by OpenAI to train GPT-2GPT-2, prop. n.: A 2019 OpenAI language model that wrote surprisingly fluent paragraphs. OpenAI at first withheld the full version, saying it could be misused for fake news, which made it famous.. Instead of filtering raw crawls with rules, it used human curation as a proxy for quality: it collected pages linked from Reddit posts that had received at least three karma, about 45 million links up to December 2017, then extracted the text, deduplicated it and removed Wikipedia articles to limit overlap with evaluation data. WebText showed that a diverse, moderately filtered web corpus could teach one model many tasks without task-specific supervision. It was never released in full (OpenAI later shared about a quarter of a million of its documents to support research on detecting generated text); the volunteer replication OpenWebText (2019) and its successor OpenWebText2, part of The PileThe Pile, prop. n.: A large free collection of English text released in 2020 by the volunteer group EleutherAI, mixing 22 sources from books and science papers to code, used to train early open language models., filled the gap, and an expanded WebText2 formed part of the training mix of GPT-3GPT-3, prop. n.: A huge 2020 OpenAI language model that could perform new tasks simply from instructions or a few examples written into its prompt, without retraining. It made prompting a new way of programming..
OpenWebText was the closest public stand-in for GPT-2's training data.
Origin
Described in 'Language Models are Unsupervised Multitask Learners' (OpenAI, February 2019) by Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei and Ilya Sutskever, the GPT-2 report. Aaron Gokaslan and Vanya Cohen released OpenWebText later in 2019.
Usage note
Using upvotes as a quality signal imports the demographics and tastes of a single platform's users. Not to be confused with generic 'web text', meaning any text taken from the web.
Related
- GPT-2GPT-2, prop. n.: A 2019 OpenAI language model that wrote surprisingly fluent paragraphs. OpenAI at first withheld the full version, saying it could be misused for fake news, which made it famous.
- GPT-3GPT-3, prop. n.: A huge 2020 OpenAI language model that could perform new tasks simply from instructions or a few examples written into its prompt, without retraining. It made prompting a new way of programming.
- pretraining datapretraining data, n.: The enormous body of text, code and other material — mostly from the web — that a large AI model first learns from, before it is tuned to follow instructions or chat.
- quality filteringquality filtering, n.: Automatically sorting through huge piles of collected text and keeping only the parts judged useful for training, such as well-written articles, while discarding spam, gibberish and junk.
- The PileThe Pile, prop. n.: A large free collection of English text released in 2020 by the volunteer group EleutherAI, mixing 22 sources from books and science papers to code, used to train early open language models.
- corpuscorpus, n.: A large collection of written or spoken language, gathered so that people or computers can study it or learn from it — like a library assembled for a language model.
Sources
- Radford, A., Wu, J., Child, R., Luan, D., Amodei, D. & Sutskever, I. (2019). Language Models are Unsupervised Multitask Learners. OpenAI technical report.
Mentioned in
The PileThe Pile, prop. n.: A large free collection of English text released in 2020 by the volunteer group EleutherAI, mixing 22 sources from books and science papers to code, used to train early open language models.