Training & Post-trainingLanguage Models & TransformersMathematics & StatisticsAGI & Superintelligence

scaling laws

nounAlso: scaling law, neural scaling laws, Kaplan scaling laws, power-law scaling

In plain wordsMeasured patterns showing that an AI model's error shrinks smoothly and predictably as the model is made bigger and given more data and computing power, so labs can forecast a giant model's performance from small, cheap experiments.

Empirical regularities, usually power lawspower law, n.: A rule linking two quantities in which scaling one up by a given factor always changes the other by the same multiple, wherever you start. Plotted on logarithmic axes, it is a straight line., that predict how much a model improves — typically measured by test cross-entropycross-entropy, n.: A score of how surprised a model's predictions are by what actually happens; lower is better. Language models are trained mainly by driving this number down on real text. loss — as more is invested in training it, whether in parameters N, training tokens D or training computetraining compute, n.: The total number of arithmetic operations performed to train a model. It is the standard yardstick for the size of an AI training effort, and some laws use it to decide which models face extra rules. C. On logarithmic axes the measured loss traces a nearly straight line that bends toward an irreducible floorirreducible loss, n.: The part of a model's error that no amount of extra size, data or training can remove, because the thing being predicted is itself partly random — like the unpredictability of the next word in an ordinary sentence.; the slopes vary with data modality and architecture, choices such as the balance of width and depth matter little, and the trends have held across many orders of magnitude. Fitted on small runs, scaling laws are used to forecast large ones; to choose model size and token count for a compute budgetcompute budget, n.: The total amount of computing a team can spend on training (or running) a model, set in advance like a money budget and used to decide how big the model should be and how much data it sees. (compute-optimal trainingcompute-optimal training, n.: Choosing a model's size and the amount of data it trains on so that a fixed amount of computing produces the best possible model — neither too big and undertrained nor too small and overfed., Chinchilla scaling lawsChinchilla scaling laws, n.: A 2022 DeepMind finding that, for a fixed amount of computing, language models had been made too big and trained on too little text; the best balance is roughly 20 word-pieces of text for every adjustable number inside the model.); to set hyperparameters such as batch sizebatch size, n.: How many training examples a model looks at before each adjustment of its settings. Bigger batches use hardware efficiently but bring diminishing returns; smaller ones are noisier but cheaper per step. and learning ratelearning rate, n.: How big a step a model takes each time it adjusts its settings during training. Too large and learning goes haywire; too small and it crawls.; and to compare architectures by their curves rather than single points. Variants cover repeated data (data-constrained scalingdata-constrained scaling, n.: How to keep improving AI models when the supply of fresh, high-quality training text runs out — for example by reusing the same data several times, which works well at first and then stops helping.), mixture of expertsmixture of experts, n.: A way to build very large AI models cheaply: the model contains many specialist sub-networks, and for each word only a few are switched on, picked by a small dispatcher network., numerical precision, downstream tasks, transfer, reinforcement learning and test-time computetest-time compute, n.: The computing effort an AI model spends while answering a question, as opposed to while being trained. Letting a model think longer or try many answers can make it markedly smarter.. They predict loss much better than specific abilities, which can seem to appear abruptly (emergent abilitiesemergent abilities, n.: Skills that seem to switch on suddenly as AI models grow: small versions fail at them, then larger ones past a certain size succeed. Some researchers argue the suddenness is partly an artifact of how skills are measured.) or worsen with scale (inverse scalinginverse scaling, n.: When bigger or more heavily trained AI models do worse on a task than smaller ones — the opposite of the usual rule that scale brings improvement.). They summarize observed trends, not laws of nature.
The scaling law fitted on runs up to 10²¹ FLOP predicted the final loss to within 1%.

L(C) \approx L_{\infty} + \left(\frac{C_0}{C}\right)^{\alpha_C}

Origin

Power-law learning curves were analyzed for neural networks in the early 1990s, and Joel Hestness and colleagues at Baidu (2017) demonstrated them across translation, language, vision and speech. The name and research program were popularized by Jared Kaplan and colleagues at OpenAI (January 2020), whose results informed GPT-3; DeepMind's Chinchilla paper (2022) revised their compute-optimal recipe.

Usage note

'Kaplan scaling laws' refers to the 2020 OpenAI fits. Whether scaling alone leads to general intelligence is a separate, contested claim: see scaling hypothesisscaling hypothesis, n.: The bet that today's AI methods, simply made bigger with more computing power and data, will keep getting smarter — possibly all the way to human-level intelligence and beyond — without needing fundamentally new ideas.. Reports in 2024–2025 that pretraining scaling was slowing concerned costs, data supply and benchmark gains; whether the underlying loss trends bent is debated.

Sources

Hestness, J. et al. (2017). Deep Learning Scaling is Predictable, Empirically. arXiv:1712.00409. arxiv.org/abs/1712.00409v1
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B. et al. (2020). Scaling Laws for Neural Language Models. arXiv:2001.08361. arxiv.org/abs/2001.08361v1
Hoffmann, J. et al. (2022). Training Compute-Optimal Large Language Models. arXiv:2203.15556. arxiv.org/abs/2203.15556v1

Mentioned in

computecompute, n.: The raw computing power used to train and run AI, usually measured by how many specialized chips are used and for how long. With data and algorithms, it is a main driver of AI progress., Dario AmodeiDario Amodei, prop. n.: An American AI researcher who co-founded Anthropic and is its chief executive, known for early work on scaling and AI safety and for forecasting very rapid AI progress., emergenceemergence, n.: When a whole shows properties or abilities that none of its parts has and that are hard to predict from them — like a traffic jam arising from individual cars, or new skills appearing in larger AI models., FLOPFLOP, n.: A single arithmetic step, such as multiplying or adding two numbers. Counting them measures how much computing went into an AI model, and laws use such counts to decide which models face extra rules., large language modellarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code., METR time horizonMETR time horizon, n.: A way of measuring AI progress by asking how long a task — timed by how long it takes a skilled human — an AI agent can complete with even odds of success., OpenAIOpenAI, prop. n.: The American company that makes ChatGPT and the GPT models. It began in 2015 as a nonprofit research lab and is now a for-profit company overseen by a nonprofit foundation., pretrainingpretraining, n.: The first and usually the most expensive stage of building a modern AI model, in which it learns general patterns from enormous amounts of raw text, images or code before being specialized for particular uses., AI timelinesAI timelines, n.: Forecasts of when AI will reach major milestones, such as matching humans at most work. People with short timelines expect this within years; others expect decades, or never., algorithmic progressalgorithmic progress, n.: Improvements in AI methods — not faster chips — that let a model reach the same ability with less computing power. Researchers estimate this has cut the computing needed by half every eight to sixteen months., BaiduBaidu, prop. n.: China's largest search-engine company and an early investor in deep learning, which makes the Ernie chatbot and models and runs driverless taxis., Chinchilla scaling lawsChinchilla scaling laws, n.: A 2022 DeepMind finding that, for a fixed amount of computing, language models had been made too big and trained on too little text; the best balance is roughly 20 word-pieces of text for every adjustable number inside the model., compute budgetcompute budget, n.: The total amount of computing a team can spend on training (or running) a model, set in advance like a money budget and used to decide how big the model should be and how much data it sees., compute thresholdcompute threshold, n.: A legal cutoff based on how much computation was used to train an AI model. Models trained with more than that amount face extra rules, on the theory that bigger training runs produce more capable models., compute-optimal trainingcompute-optimal training, n.: Choosing a model's size and the amount of data it trains on so that a fixed amount of computing produces the best possible model — neither too big and undertrained nor too small and overfed., convergenceconvergence, n.: The point at which training settles down and further steps no longer meaningfully improve the model — and, in theory, a guarantee that a method will reach a solution, and how fast., data walldata wall, n.: The worry that AI companies will run out of new human-written text to train ever-larger models on, because the useful public internet has already been largely used up., emergent abilitiesemergent abilities, n.: Skills that seem to switch on suddenly as AI models grow: small versions fail at them, then larger ones past a certain size succeed. Some researchers argue the suddenness is partly an artifact of how skills are measured., Epoch AIEpoch AI, prop. n.: A research nonprofit that gathers data on how fast AI is advancing — the computing power, data and money behind it — and publishes forecasts and difficult tests for AI., GPT-3GPT-3, prop. n.: A huge 2020 OpenAI language model that could perform new tasks simply from instructions or a few examples written into its prompt, without retraining. It made prompting a new way of programming., hyperparameterhyperparameter, n.: A setting chosen by people before training, such as how fast a model learns or how big it is, as opposed to the internal values the model learns by itself from data., intelligence explosionintelligence explosion, n.: The idea that once AI gets good enough at improving AI, each smarter version could build a smarter one faster still, so abilities would snowball far beyond human level in a short time., Jared KaplanJared Kaplan, prop. n.: An American physicist who showed that language models improve predictably as they grow and train on more data, and who co-founded the AI company Anthropic., learning curvelearning curve, n.: A graph showing how a model's performance improves as it gets more training — more examples or more time. Its shape tells you whether more data or a bigger model would help.

scaling laws

Origin

Usage note

Related

Sources

Mentioned in