underfitting
In plain wordsWhen a model is too simple, or trained too little, to capture the real pattern in the data, so it does poorly even on the examples it learned from.
The failure in which a model does not represent even the regularities present in its own training data, so that its error is high on the training set and about as high on new data. Causes include insufficient capacity (a straight line fit to a curved relationship), overly strong regularizationregularization, n.: Any technique that stops a model from memorizing its training examples too closely — often by penalizing complexity — so that it works better on new data., uninformative or too few featuresfeature, n.: One piece of information about each example that a model uses, like a house's size or a pixel's color, or a pattern a model has learned to detect inside itself., and stopping training too early. It corresponds to the high-bias side of the bias–variance tradeoffbias–variance tradeoff, n.: The balancing act between models too simple to capture a pattern and models so flexible that they also learn the noise. The best predictions usually come from somewhere in between. and is recognized by training and validation errors that are both high and close together, often with a learning curvelearning curve, n.: A graph showing how a model's performance improves as it gets more training — more examples or more time. Its shape tells you whether more data or a bigger model would help. that has plateaued. Remedies are the reverse of those for overfittingoverfitting, n.: When a model learns its training examples too well — quirks and noise included — so it does great on data it has seen but poorly on anything new, like a student who memorizes answers without understanding.: more flexible models, richer features, weaker regularization or longer training. Large language models are typically pretrained in a regime where they see most data only once and loss keeps falling with more data and compute along scaling lawsscaling laws, n.: Measured patterns showing that an AI model's error shrinks smoothly and predictably as the model is made bigger and given more data and computing power, so labs can forecast a giant model's performance from small, cheap experiments., so overall they sit nearer the underfitting side, though they can still memorize passages repeated in their data.
Both training and validation loss plateaued at 0.9, so the model is underfitting; add capacity before adding regularization.
Origin
Formed as the counterpart of 'overfitting' in statistics and machine learning; its first use has not been established, but the paired terms were common in textbook treatments of model complexity by the 1990s.
Related
- overfittingoverfitting, n.: When a model learns its training examples too well — quirks and noise included — so it does great on data it has seen but poorly on anything new, like a student who memorizes answers without understanding.
- bias–variance tradeoffbias–variance tradeoff, n.: The balancing act between models too simple to capture a pattern and models so flexible that they also learn the noise. The best predictions usually come from somewhere in between.
- learning curvelearning curve, n.: A graph showing how a model's performance improves as it gets more training — more examples or more time. Its shape tells you whether more data or a bigger model would help.
- regularizationregularization, n.: Any technique that stops a model from memorizing its training examples too closely — often by penalizing complexity — so that it works better on new data.
- model selectionmodel selection, n.: Choosing, from several candidate models or settings, the one most likely to work best on new data, rather than the one that merely fits the existing data best.
- scaling lawsscaling laws, n.: Measured patterns showing that an AI model's error shrinks smoothly and predictably as the model is made bigger and given more data and computing power, so labs can forecast a giant model's performance from small, cheap experiments.
- overfittingoverfitting, n.: When a model learns its training examples too well — quirks and noise included — so it does great on data it has seen but poorly on anything new, like a student who memorizes answers without understanding.
Mentioned in
overfittingoverfitting, n.: When a model learns its training examples too well — quirks and noise included — so it does great on data it has seen but poorly on anything new, like a student who memorizes answers without understanding., bias–variance tradeoffbias–variance tradeoff, n.: The balancing act between models too simple to capture a pattern and models so flexible that they also learn the noise. The best predictions usually come from somewhere in between., decision boundarydecision boundary, n.: The dividing line, or surface, where a classifier switches from predicting one category to another. Simple models draw straight lines; flexible ones draw curves., generalization gapgeneralization gap, n.: The difference between how well a model does on the examples it trained on and how well it does on new ones. A big gap means it memorized rather than learned., learning curvelearning curve, n.: A graph showing how a model's performance improves as it gets more training — more examples or more time. Its shape tells you whether more data or a bigger model would help., model capacitymodel capacity, n.: How much a model is able to learn: the variety and complexity of patterns it can represent or memorize. Too little and it misses real patterns; very large models can even memorize pure noise., polynomial regressionpolynomial regression, n.: Fitting a curve instead of a straight line by also using the square, cube and higher powers of an input. Higher powers let the curve bend more but easily start chasing noise., hypothesis spacehypothesis space, n.: The full menu of possible models a learning method is allowed to choose from — for example, all straight lines. Learning means picking the item on the menu that best fits the data.