irreducible loss
In plain wordsThe part of a model's error that no amount of extra size, data or training can remove, because the thing being predicted is itself partly random — like the unpredictability of the next word in an ordinary sentence.
- 1
Scaling laws
The constant term E (or L∞) in a scaling-law fit such as L = E + A/N^α + B/D^β: the loss that would remain with unlimited parameters, data and compute. For cross-entropycross-entropy, n.: A score of how surprised a model's predictions are by what actually happens; lower is better. Language models are trained mainly by driving this number down on real text. loss it is interpreted as the entropyentropy, n.: A measure of unpredictability. A fair coin toss has more entropy than a rigged one; an AI that is unsure of its next word has high entropy. of the data-generating distribution — the inherent unpredictability of the text, images or audio — while the remaining reducible loss corresponds to the Kullback–Leibler divergenceKullback–Leibler divergence, n.: A score for the cost of believing the wrong odds: on average, how much more surprised you will be by real outcomes than someone who knew the true probabilities. It is zero only when your odds are exactly right. between that distribution and the model's. Estimates depend on the dataset and tokenizertokenizer, n.: The component that splits text into tokens and converts them to numbers before an AI model sees it, then turns the model's output numbers back into text. and are extrapolations, not measurements: the ChinchillaChinchilla scaling laws, n.: A 2022 DeepMind finding that, for a fixed amount of computing, language models had been made too big and trained on too little text; the best balance is roughly 20 word-pieces of text for every adjustable number inside the model. fit gave E ≈ 1.69 nats per token on DeepMind's MassiveText data, and a 2024 refit about 1.82. Because only the reducible part can shrink, progress is better judged against it than against total loss.
- 2
Data selection
In reducible holdout loss selection (RHO-LOSS, Mindermann et al., 2022), the loss that a small model trained on held-out data assigns to an example. Training prioritizes examples whose current loss greatly exceeds this value, skipping points that are noisy or unlearnable rather than merely not yet learned.
Origin
The idea descends from the Bayes error and the noise term of the classical bias–variance decompositionbias–variance tradeoff, n.: The balancing act between models too simple to capture a pattern and models so flexible that they also learn the noise. The best predictions usually come from somewhere in between.. For deep-learning scaling, Joel Hestness and colleagues (2017) described learning curves flattening into an irreducible-error region, and Tom Henighan, Jared Kaplan and colleagues (2020) read the constant in power-law-plus-constant fits as the data's entropy.
Related
- scaling lawsscaling laws, n.: Measured patterns showing that an AI model's error shrinks smoothly and predictably as the model is made bigger and given more data and computing power, so labs can forecast a giant model's performance from small, cheap experiments.
- entropyentropy, n.: A measure of unpredictability. A fair coin toss has more entropy than a rigged one; an AI that is unsure of its next word has high entropy.
- cross-entropycross-entropy, n.: A score of how surprised a model's predictions are by what actually happens; lower is better. Language models are trained mainly by driving this number down on real text.
- Kullback–Leibler divergenceKullback–Leibler divergence, n.: A score for the cost of believing the wrong odds: on average, how much more surprised you will be by real outcomes than someone who knew the true probabilities. It is zero only when your odds are exactly right.
- Chinchilla scaling lawsChinchilla scaling laws, n.: A 2022 DeepMind finding that, for a fixed amount of computing, language models had been made too big and trained on too little text; the best balance is roughly 20 word-pieces of text for every adjustable number inside the model.
- power lawpower law, n.: A rule linking two quantities in which scaling one up by a given factor always changes the other by the same multiple, wherever you start. Plotted on logarithmic axes, it is a straight line.
- bias–variance tradeoffbias–variance tradeoff, n.: The balancing act between models too simple to capture a pattern and models so flexible that they also learn the noise. The best predictions usually come from somewhere in between.
Sources
- Henighan, T., Kaplan, J., Katz, M., Chen, M. et al. (2020). Scaling Laws for Autoregressive Generative Modeling. arXiv:2010.14701. arxiv.org/abs/2010.14701
- Mindermann, S. et al. (2022). Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learnt. ICML 2022.
Mentioned in
scaling lawsscaling laws, n.: Measured patterns showing that an AI model's error shrinks smoothly and predictably as the model is made bigger and given more data and computing power, so labs can forecast a giant model's performance from small, cheap experiments., Chinchilla scaling lawsChinchilla scaling laws, n.: A 2022 DeepMind finding that, for a fixed amount of computing, language models had been made too big and trained on too little text; the best balance is roughly 20 word-pieces of text for every adjustable number inside the model., power lawpower law, n.: A rule linking two quantities in which scaling one up by a given factor always changes the other by the same multiple, wherever you start. Plotted on logarithmic axes, it is a straight line., Bayes error rateBayes error rate, n.: The lowest error any classifier could ever achieve on a task, because some cases are genuinely ambiguous: even a perfect model that knew the true odds would still sometimes guess wrong.