early stopping
In plain wordsStopping training at the point where the model does best on practice data it has not learned from, before it starts memorizing quirks of its training examples.
A regularizationregularization, n.: Any technique that stops a model from memorizing its training examples too closely — often by penalizing complexity — so that it works better on new data. method that monitors performance on a held-out validation setvalidation set, n.: A set of examples kept out of training and used during development to compare model versions and settings, like a practice exam taken before the real one. during training and halts — or keeps the best checkpointcheckpoint, n.: A saved snapshot of a model partway through or at the end of training, like a save point in a video game: training can resume from it after a crash, and finished snapshots are what people download and run. — once the validation metric has stopped improving for a set number of evaluations (the 'patience'), even though training loss is still falling. It exploits the typical shape of learning curveslearning curve, n.: A graph showing how a model's performance improves as it gets more training — more examples or more time. Its shape tells you whether more data or a bigger model would help.: validation error first falls with training error and later rises as the model starts overfittingoverfitting, n.: When a model learns its training examples too well — quirks and noise included — so it does great on data it has seen but poorly on anything new, like a student who memorizes answers without understanding.. For linear models trained by gradient descent, stopping early behaves much like an L2 penalty, because the number of steps limits how far parameters can travel from their initialization; for neural networks it is a cheap capacity control that also saves compute. Noisy validation metrics can trigger it too soon, and phenomena such as epoch-wise double descentdouble descent, n.: A surprising pattern where a model's error on new data first falls, then rises, then falls again as the model grows bigger — contradicting the old rule that oversized models always do worse. and grokkinggrokking, n.: When a model that has merely memorized its training examples suddenly, long afterward, figures out the underlying rule and starts getting new examples right., in which validation performance worsens before improving again, mean naive patience can stop before the best model is reached.
Early stopping with patience 5 picked the checkpoint from epoch 23.
Origin
Halting iterative methods early as a form of regularization predates neural networks, for example in iterative solutions of ill-posed inverse problems. In neural-network training it was in use by about 1990 (for instance by Nelson Morgan and Hervé Bourlard) and was systematized in Lutz Prechelt's chapter 'Early Stopping — But When?' (1998).
Usage note
Large language model pretrainingpretraining, n.: The first and usually the most expensive stage of building a modern AI model, in which it learns general patterns from enormous amounts of raw text, images or code before being specialized for particular uses. rarely uses early stopping: models see most data about once and rarely overfit, so runs end when the compute budget is spent.
Related
- overfittingoverfitting, n.: When a model learns its training examples too well — quirks and noise included — so it does great on data it has seen but poorly on anything new, like a student who memorizes answers without understanding.
- validation setvalidation set, n.: A set of examples kept out of training and used during development to compare model versions and settings, like a practice exam taken before the real one.
- learning curvelearning curve, n.: A graph showing how a model's performance improves as it gets more training — more examples or more time. Its shape tells you whether more data or a bigger model would help.
- regularizationregularization, n.: Any technique that stops a model from memorizing its training examples too closely — often by penalizing complexity — so that it works better on new data.
- checkpointcheckpoint, n.: A saved snapshot of a model partway through or at the end of training, like a save point in a video game: training can resume from it after a crash, and finished snapshots are what people download and run.
- double descentdouble descent, n.: A surprising pattern where a model's error on new data first falls, then rises, then falls again as the model grows bigger — contradicting the old rule that oversized models always do worse.
Sources
- Prechelt, L. (1998). Early Stopping — But When? In Neural Networks: Tricks of the Trade, Lecture Notes in Computer Science 1524, 55–69.
Mentioned in
overfittingoverfitting, n.: When a model learns its training examples too well — quirks and noise included — so it does great on data it has seen but poorly on anything new, like a student who memorizes answers without understanding., convergenceconvergence, n.: The point at which training settles down and further steps no longer meaningfully improve the model — and, in theory, a guarantee that a method will reach a solution, and how fast., epochepoch, n.: One complete pass through all the training examples. Smaller models are often trained for many epochs, seeing each example many times; the largest language models may see most of their data only once., gradient boostinggradient boosting, n.: A method that builds a strong predictor by adding small decision trees one at a time, each trained to correct the errors that remain after all the trees before it., learning curvelearning curve, n.: A graph showing how a model's performance improves as it gets more training — more examples or more time. Its shape tells you whether more data or a bigger model would help., overtrainingovertraining, n.: Deliberately training a model on much more data than would be most efficient for its training budget, so that a smaller model reaches good quality and is cheaper to run afterwards., regularizationregularization, n.: Any technique that stops a model from memorizing its training examples too closely — often by penalizing complexity — so that it works better on new data., training looptraining loop, n.: The repeating cycle at the heart of training software: grab some examples, let the model guess, measure the error, adjust the model, and do it again — often millions of times., validation setvalidation set, n.: A set of examples kept out of training and used during development to compare model versions and settings, like a practice exam taken before the real one., early exitearly exit, n.: Letting an AI model stop partway through its layers and answer early when it is already confident, saving computing effort on easy inputs., grokkinggrokking, n.: When a model that has merely memorized its training examples suddenly, long afterward, figures out the underlying rule and starts getting new examples right., HyperbandHyperband, n.: A tuning method that starts many candidate settings on small budgets, keeps only the most promising ones, and gives the survivors more training time, so effort isn't wasted on losers., implicit regularizationimplicit regularization, n.: The hidden preferences of a training method itself: even with no explicit rule against complexity, the way a model is trained tends to steer it toward simpler solutions.