data mixture
In plain wordsThe recipe of different kinds of data a model is trained on — how much web text, code, books, math, science and other languages — which strongly shapes what the finished model is good at.
The set of data sources used to train a model together with the proportions in which each is sampled — for example web pages, code, books, scientific papers, mathematics, conversational data and many languages. Because sampling weights need not match source sizes, small high-value sources are often upsampled (seen for several epochsepoch, n.: One complete pass through all the training examples. Smaller models are often trained for many epochs, seeing each example many times; the largest language models may see most of their data only once.) and large noisy ones downsampled: GPT-3GPT-3, prop. n.: A huge 2020 OpenAI language model that could perform new tasks simply from instructions or a few examples written into its prompt, without retraining. It made prompting a new way of programming., for instance, drew 60% of its training tokens from filtered Common CrawlCommon Crawl, prop. n.: A nonprofit's free archive of the public web, begun in 2008 and now refreshed with snapshots of billions of pages every month or two. Most large language models learned much of what they know from text taken from it. but sampled Wikipedia several times over. Mixtures are tuned by heuristics and ablations on small proxy models, by learned reweighting such as DoReMi (2023), and by fitted data-mixing laws that predict loss from proportions. Modern recipes change the mixture during a run, upweighting code, mathematics, reasoning traces and synthetic data in annealingannealing, n.: The last stretch of training, when the model's learning is slowed to a stop — often while it is fed a small helping of the best data — so it settles into its final, sharpest version. or mid-trainingmid-training, n.: A middle stage in building a language model, between its broad initial training and its tuning into an assistant: it keeps learning the same way, but from a smaller, carefully chosen diet of math, code, reasoning or long documents.. Mixture details are among the most closely guarded parts of frontier recipes.
Raising code from 10% to 25% of the data mixture improved reasoning benchmarks but hurt multilingual ones.
Origin
Weighted combinations of corpora are old in language modeling; the modern framing comes from 2020–2021 work such as GPT-3's sampling weights, EleutherAI's The Pile (22 subsets) and DeepMind's MassiveText, followed by automated methods such as Sang Michael Xie and colleagues' DoReMi (2023).
Usage note
Unrelated to mixture of expertsmixture of experts, n.: A way to build very large AI models cheaply: the model contains many specialist sub-networks, and for each word only a few are switched on, picked by a small dispatcher network., an architecture. Mixture effects interact with quality filteringquality filtering, n.: Automatically sorting through huge piles of collected text and keeping only the parts judged useful for training, such as well-written articles, while discarding spam, gibberish and junk. and deduplicationdeduplication, n.: Removing repeated copies of the same or nearly the same content from a dataset, so a model does not over-learn whatever happens to be copied many times online., so the same nominal proportions can yield different models.
Related
- pretraining datapretraining data, n.: The enormous body of text, code and other material — mostly from the web — that a large AI model first learns from, before it is tuned to follow instructions or chat.
- annealingannealing, n.: The last stretch of training, when the model's learning is slowed to a stop — often while it is fed a small helping of the best data — so it settles into its final, sharpest version.
- mid-trainingmid-training, n.: A middle stage in building a language model, between its broad initial training and its tuning into an assistant: it keeps learning the same way, but from a smaller, carefully chosen diet of math, code, reasoning or long documents.
- quality filteringquality filtering, n.: Automatically sorting through huge piles of collected text and keeping only the parts judged useful for training, such as well-written articles, while discarding spam, gibberish and junk.
- The PileThe Pile, prop. n.: A large free collection of English text released in 2020 by the volunteer group EleutherAI, mixing 22 sources from books and science papers to code, used to train early open language models.
- Common CrawlCommon Crawl, prop. n.: A nonprofit's free archive of the public web, begun in 2008 and now refreshed with snapshots of billions of pages every month or two. Most large language models learned much of what they know from text taken from it.
- data-constrained scalingdata-constrained scaling, n.: How to keep improving AI models when the supply of fresh, high-quality training text runs out — for example by reusing the same data several times, which works well at first and then stops helping.
- mixture of expertsmixture of experts, n.: A way to build very large AI models cheaply: the model contains many specialist sub-networks, and for each word only a few are switched on, picked by a small dispatcher network.
Sources
- Brown, T. B. et al. (2020). Language Models are Few-Shot Learners. NeurIPS 2020.
- Xie, S. M. et al. (2023). DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining. NeurIPS 2023.
Mentioned in
pretrainingpretraining, n.: The first and usually the most expensive stage of building a modern AI model, in which it learns general patterns from enormous amounts of raw text, images or code before being specialized for particular uses., annealingannealing, n.: The last stretch of training, when the model's learning is slowed to a stop — often while it is fed a small helping of the best data — so it settles into its final, sharpest version., curriculum learningcurriculum learning, n.: Teaching a model the way schools teach people: starting with easy examples and gradually moving to harder ones, instead of presenting everything in random order., data pipelinedata pipeline, n.: The chain of automated steps that collects raw data, cleans and transforms it, and delivers it in a usable form — the factory line that turns messy inputs into training material., mid-trainingmid-training, n.: A middle stage in building a language model, between its broad initial training and its tuning into an assistant: it keeps learning the same way, but from a smaller, carefully chosen diet of math, code, reasoning or long documents., pretraining datapretraining data, n.: The enormous body of text, code and other material — mostly from the web — that a large AI model first learns from, before it is tuned to follow instructions or chat., training runtraining run, n.: One complete attempt at training a model with a fixed recipe, from start to finish. For the largest AI models a single run can occupy tens of thousands of chips for months., quality filteringquality filtering, n.: Automatically sorting through huge piles of collected text and keeping only the parts judged useful for training, such as well-written articles, while discarding spam, gibberish and junk., The PileThe Pile, prop. n.: A large free collection of English text released in 2020 by the volunteer group EleutherAI, mixing 22 sources from books and science papers to code, used to train early open language models.