data-constrained scaling
In plain wordsHow to keep improving AI models when the supply of fresh, high-quality training text runs out — for example by reusing the same data several times, which works well at first and then stops helping.
The study of scaling lawsscaling laws, n.: Measured patterns showing that an AI model's error shrinks smoothly and predictably as the model is made bigger and given more data and computing power, so labs can forecast a giant model's performance from small, cheap experiments. and compute allocation when unique training data are limited, so that extra compute must go to repeated epochsepoch, n.: One complete pass through all the training examples. Smaller models are often trained for many epochs, seeing each example many times; the largest language models may see most of their data only once., larger models or substitute data. Training more than 400 models, with up to 9 billion parameters and 900 billion tokens, Niklas Muennighoff and colleagues (2023) found that at a fixed compute budget, repeating data for up to about four epochs costs almost nothing compared with fresh data, but each further pass is worth less, until more repetition adds essentially nothing. Their fitted law gives diminishing weight to repeated tokens and to parameters beyond the optimum, and implies that under scarcity compute should go to more epochs before larger models. Mixing in code served as a substitute for scarce text (up to half the data, with no loss on natural-language tasks), while the value of strict quality filteringquality filtering, n.: Automatically sorting through huge piles of collected text and keeping only the parts judged useful for training, such as well-written articles, while discarding spam, gibberish and junk. depended on how noisy the data were; synthetic datasynthetic data, n.: Data created by computer programs or AI models rather than collected from the real world — such as practice math problems written by a model, or simulated street scenes for self-driving cars. is the other main lever. The question grew pressing as Epoch AIEpoch AI, prop. n.: A research nonprofit that gathers data on how fast AI is advancing — the computing power, data and money behind it — and publishes forecasts and difficult tests for AI. projected that frontier training could exhaust the stock of public human-written text between about 2026 and 2032 (the data walldata wall, n.: The worry that AI companies will run out of new human-written text to train ever-larger models on, because the useful public internet has already been largely used up.).
Under data-constrained scaling, four epochs over our 2-trillion-token corpus cost us almost nothing in loss.
Origin
Named by Niklas Muennighoff, Alexander M. Rush, Boaz Barak, Teven Le Scao, Colin Raffel and colleagues in 'Scaling Data-Constrained Language Models' (arXiv, May 2023; NeurIPS 2023), which extended the Chinchilla framework to repeated data.
Usage note
Concerns repetition of the same data, not low-quality data. The results are for pretrainingpretraining, n.: The first and usually the most expensive stage of building a modern AI model, in which it learns general patterns from enormous amounts of raw text, images or code before being specialized for particular uses.; small high-quality sets used in annealingannealing, n.: The last stretch of training, when the model's learning is slowed to a stop — often while it is fed a small helping of the best data — so it settles into its final, sharpest version. or mid-trainingmid-training, n.: A middle stage in building a language model, between its broad initial training and its tuning into an assistant: it keeps learning the same way, but from a smaller, carefully chosen diet of math, code, reasoning or long documents. are often repeated more often.
Related
- data walldata wall, n.: The worry that AI companies will run out of new human-written text to train ever-larger models on, because the useful public internet has already been largely used up.
- scaling lawsscaling laws, n.: Measured patterns showing that an AI model's error shrinks smoothly and predictably as the model is made bigger and given more data and computing power, so labs can forecast a giant model's performance from small, cheap experiments.
- epochepoch, n.: One complete pass through all the training examples. Smaller models are often trained for many epochs, seeing each example many times; the largest language models may see most of their data only once.
- synthetic datasynthetic data, n.: Data created by computer programs or AI models rather than collected from the real world — such as practice math problems written by a model, or simulated street scenes for self-driving cars.
- Chinchilla scaling lawsChinchilla scaling laws, n.: A 2022 DeepMind finding that, for a fixed amount of computing, language models had been made too big and trained on too little text; the best balance is roughly 20 word-pieces of text for every adjustable number inside the model.
- tokens per parametertokens per parameter, n.: How many pieces of text a language model reads during training for each adjustable number inside it — a quick way to tell whether a model was trained on a little or a lot of data for its size.
Sources
- Muennighoff, N., Rush, A. M., Barak, B., Le Scao, T. et al. (2023). Scaling Data-Constrained Language Models. arXiv:2305.16264. arxiv.org/abs/2305.16264
Mentioned in
scaling lawsscaling laws, n.: Measured patterns showing that an AI model's error shrinks smoothly and predictably as the model is made bigger and given more data and computing power, so labs can forecast a giant model's performance from small, cheap experiments., data mixturedata mixture, n.: The recipe of different kinds of data a model is trained on — how much web text, code, books, math, science and other languages — which strongly shapes what the finished model is good at., data walldata wall, n.: The worry that AI companies will run out of new human-written text to train ever-larger models on, because the useful public internet has already been largely used up., epochepoch, n.: One complete pass through all the training examples. Smaller models are often trained for many epochs, seeing each example many times; the largest language models may see most of their data only once., overtrainingovertraining, n.: Deliberately training a model on much more data than would be most efficient for its training budget, so that a smaller model reaches good quality and is cheaper to run afterwards., tokens per parametertokens per parameter, n.: How many pieces of text a language model reads during training for each adjustable number inside it — a quick way to tell whether a model was trained on a little or a lot of data for its size.