overparameterization
In plain wordsGiving a model far more adjustable settings than it would seem to need — often more than it has training examples. Classic theory predicted this would backfire; in deep learning it usually helps.
The condition in which a model has more parameters than needed to fit its training data exactly — often more parameters than training examples — so that infinitely many settings achieve zero training loss. Classical statistics associated this with severe overfittingoverfitting, n.: When a model learns its training examples too well — quirks and noise included — so it does great on data it has seen but poorly on anything new, like a student who memorizes answers without understanding., yet overparameterized neural networks routinely generalize well, can fit even randomly labeled data, and often improve as they grow. Proposed explanations include optimization benefits (wider networks have more benign loss landscapesloss landscape, n.: A way of picturing training as a journey across hilly terrain, where position stands for the model's settings and height for its error. Training searches for low valleys. and, in the neural tangent kernelneural tangent kernel, n.: A mathematical tool showing that extremely wide neural networks learn like a simpler, well-understood method, because their internal settings barely move during training. It explains some, but not all, of deep learning's behavior. regime, provably reach zero training loss); implicit regularizationimplicit regularization, n.: The hidden preferences of a training method itself: even with no explicit rule against complexity, the way a model is trained tends to steer it toward simpler solutions. by gradient descentgradient descent, n.: A method for training models by repeatedly nudging every setting a small step in whichever direction most reduces the error, like a hiker who cannot see the valley but keeps stepping wherever the ground tilts down., which selects simple, low-norm interpolating solutions; the double descentdouble descent, n.: A surprising pattern where a model's error on new data first falls, then rises, then falls again as the model grows bigger — contradicting the old rule that oversized models always do worse. curve beyond the interpolation threshold; and benign overfittingbenign overfitting, n.: When a model fits its training data perfectly — even the errors and noise — yet still predicts well on new data, defying the textbook warning that this always backfires. theory, which shows when fitting noise does little harm. The lottery ticket hypothesislottery ticket hypothesis, n.: The idea that a large, randomly started neural network contains a much smaller network inside it that, trained alone from the same starting values, would learn just as well — a winning lottery ticket. adds that extra parameters supply many candidate subnetworks. Large language models reverse the usual ratio, typically training on far more tokens than they have parameters, so the label fits them less well than it fits image classifiers.
Origin
The term comes from statistics and system identification, where an overparameterized model has more parameters than the data can determine. Its rehabilitation in deep learning dates to the 2010s: Behnam Neyshabur and colleagues (2014) and Chiyuan Zhang and colleagues' 'Understanding deep learning requires rethinking generalization' (2016), which showed networks memorizing random labels, set the agenda for later theory.
Related
- double descentdouble descent, n.: A surprising pattern where a model's error on new data first falls, then rises, then falls again as the model grows bigger — contradicting the old rule that oversized models always do worse.
- benign overfittingbenign overfitting, n.: When a model fits its training data perfectly — even the errors and noise — yet still predicts well on new data, defying the textbook warning that this always backfires.
- implicit regularizationimplicit regularization, n.: The hidden preferences of a training method itself: even with no explicit rule against complexity, the way a model is trained tends to steer it toward simpler solutions.
- neural tangent kernelneural tangent kernel, n.: A mathematical tool showing that extremely wide neural networks learn like a simpler, well-understood method, because their internal settings barely move during training. It explains some, but not all, of deep learning's behavior.
- lottery ticket hypothesislottery ticket hypothesis, n.: The idea that a large, randomly started neural network contains a much smaller network inside it that, trained alone from the same starting values, would learn just as well — a winning lottery ticket.
- overfittingoverfitting, n.: When a model learns its training examples too well — quirks and noise included — so it does great on data it has seen but poorly on anything new, like a student who memorizes answers without understanding.
- parameter countparameter count, n.: How many adjustable numbers an AI model contains — the usual measure of its size. Modern language models have billions to trillions of them.
Sources
- Zhang, C., Bengio, S., Hardt, M., Recht, B. & Vinyals, O. (2017). Understanding deep learning requires rethinking generalization. ICLR 2017. arxiv.org/abs/1611.03530
Mentioned in
generalizationgeneralization, n.: How well a model copes with new examples it has never seen, rather than just memorizing its training data, like a student who understands the method, not just the answers., local minimumlocal minimum, n.: A spot lower than everything immediately around it, like the bottom of a small valley, even though a deeper valley may exist somewhere else., model capacitymodel capacity, n.: How much a model is able to learn: the variety and complexity of patterns it can represent or memorize. Too little and it misses real patterns; very large models can even memorize pure noise., Occam's razorOccam's razor, n.: A rule of thumb for choosing between explanations: if two fit the evidence equally well, bet on the one that assumes less. In AI it motivates preferring simpler models, which are less likely to fit noise., parameterparameter, n.: One of the adjustable numbers inside an AI model that are tuned during training. Large language models have billions of them, and together they store what the model has learned., benign overfittingbenign overfitting, n.: When a model fits its training data perfectly — even the errors and noise — yet still predicts well on new data, defying the textbook warning that this always backfires., double descentdouble descent, n.: A surprising pattern where a model's error on new data first falls, then rises, then falls again as the model grows bigger — contradicting the old rule that oversized models always do worse., implicit regularizationimplicit regularization, n.: The hidden preferences of a training method itself: even with no explicit rule against complexity, the way a model is trained tends to steer it toward simpler solutions., lottery ticket hypothesislottery ticket hypothesis, n.: The idea that a large, randomly started neural network contains a much smaller network inside it that, trained alone from the same starting values, would learn just as well — a winning lottery ticket., neural tangent kernelneural tangent kernel, n.: A mathematical tool showing that extremely wide neural networks learn like a simpler, well-understood method, because their internal settings barely move during training. It explains some, but not all, of deep learning's behavior., VC dimensionVC dimension, n.: A measure of how flexible a family of prediction rules is: the largest number of points it can label in every possible way. More flexible families need more data to learn reliably.