learning rate schedule
In plain wordsA plan for changing how big a model's learning steps are during training — usually starting cautiously, then taking big steps, then smaller and smaller ones as it settles in.
A rule that sets the learning ratelearning rate, n.: How big a step a model takes each time it adjusts its settings during training. Too large and learning goes haywire; too small and it crawls. as a function of the training steptraining step, n.: One round of learning: the model makes predictions on a small batch of examples, measures its error and adjusts its settings once. Training runs consist of thousands to millions of steps. or epochepoch, n.: One complete pass through all the training examples. Smaller models are often trained for many epochs, seeing each example many times; the largest language models may see most of their data only once. instead of holding it fixed. Common forms are step decay (divide by 10 at set epochs, standard for ImageNet-era convolutional networks), exponential and polynomial decay, inverse-square-root decay (used by the original transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it.), cosine annealingcosine annealing, n.: A popular plan for shrinking a model's step size during training: steps start large, shrink slowly, then faster, then level off near zero, following the shape of half a wave., and the warmup-stable-decay schedulewarmup-stable-decay schedule, n.: A plan for how fast a model learns during training: speed up briefly, hold steady for most of the run, then slow down sharply at the end — which lets labs stop or extend training without starting over., which holds the rate constant and decays only near the end; almost all modern schedules begin with learning rate warmuplearning rate warmup, n.: Starting training with tiny learning steps and gradually increasing them over the first stretch of training, so the model does not lurch wildly before it has found its footing.. Decay lets an optimizer progress quickly with large steps early, then reduce the noise of stochastic gradient descentstochastic gradient descent, n.: A faster version of gradient descent that estimates which way is downhill from a small random sample of the training data at each step, instead of checking the whole dataset. so it can settle into a low-loss region, which is why the loss often drops sharply when the rate is cut. Schedules that need the total training length in advance complicate open-ended or continued training, motivating extendable schedules and schedule-free optimizationschedule-free optimization, n.: A way of training that needs no plan for slowing learning down over time, so training can be stopped or extended at any point and still produce a well-settled model.. 'Annealing' is used loosely for any decay and also names a data-focused final phase of LLM pretraining (see annealingannealing, n.: The last stretch of training, when the model's learning is slowed to a stop — often while it is fed a small helping of the best data — so it settles into its final, sharpest version.).
Switching from step decay to a cosine schedule gained half a point of top-1 accuracy.
Origin
Decreasing step sizes are as old as stochastic approximation (Herbert Robbins and Sutton Monro, 1951), whose convergence proofs require them. Hand-tuned decay was routine in neural-network training by the 1990s; cosine annealing (2016), warmup with inverse-square-root decay (2017) and warmup-stable-decay (2024) are later landmarks.
Usage note
Comparisons between optimizers are sensitive to the schedule each is given, so benchmarks such as AlgoPerf tune or fix schedules explicitly.
Related
- learning ratelearning rate, n.: How big a step a model takes each time it adjusts its settings during training. Too large and learning goes haywire; too small and it crawls.
- learning rate warmuplearning rate warmup, n.: Starting training with tiny learning steps and gradually increasing them over the first stretch of training, so the model does not lurch wildly before it has found its footing.
- cosine annealingcosine annealing, n.: A popular plan for shrinking a model's step size during training: steps start large, shrink slowly, then faster, then level off near zero, following the shape of half a wave.
- warmup-stable-decay schedulewarmup-stable-decay schedule, n.: A plan for how fast a model learns during training: speed up briefly, hold steady for most of the run, then slow down sharply at the end — which lets labs stop or extend training without starting over.
- schedule-free optimizationschedule-free optimization, n.: A way of training that needs no plan for slowing learning down over time, so training can be stopped or extended at any point and still produce a well-settled model.
- annealingannealing, n.: The last stretch of training, when the model's learning is slowed to a stop — often while it is fed a small helping of the best data — so it settles into its final, sharpest version.
Mentioned in
pretrainingpretraining, n.: The first and usually the most expensive stage of building a modern AI model, in which it learns general patterns from enormous amounts of raw text, images or code before being specialized for particular uses., AdamWAdamW, prop. n.: A corrected version of the popular Adam training method that applies its steady pull toward smaller weights separately from the main update. It is the standard choice for training large language models., annealingannealing, n.: The last stretch of training, when the model's learning is slowed to a stop — often while it is fed a small helping of the best data — so it settles into its final, sharpest version., compute-optimal trainingcompute-optimal training, n.: Choosing a model's size and the amount of data it trains on so that a fixed amount of computing produces the best possible model — neither too big and undertrained nor too small and overfed., convergenceconvergence, n.: The point at which training settles down and further steps no longer meaningfully improve the model — and, in theory, a guarantee that a method will reach a solution, and how fast., cosine annealingcosine annealing, n.: A popular plan for shrinking a model's step size during training: steps start large, shrink slowly, then faster, then level off near zero, following the shape of half a wave., epochepoch, n.: One complete pass through all the training examples. Smaller models are often trained for many epochs, seeing each example many times; the largest language models may see most of their data only once., learning ratelearning rate, n.: How big a step a model takes each time it adjusts its settings during training. Too large and learning goes haywire; too small and it crawls., learning rate warmuplearning rate warmup, n.: Starting training with tiny learning steps and gradually increasing them over the first stretch of training, so the model does not lurch wildly before it has found its footing., loss curveloss curve, n.: A graph of a model's error as training goes on. A healthy curve drops steeply and then keeps sloping gently downward; bumps, jumps or flat stretches can warn engineers that something is wrong., stochastic gradient descentstochastic gradient descent, n.: A faster version of gradient descent that estimates which way is downhill from a small random sample of the training data at each step, instead of checking the whole dataset., training looptraining loop, n.: The repeating cycle at the heart of training software: grab some examples, let the model guess, measure the error, adjust the model, and do it again — often millions of times., training steptraining step, n.: One round of learning: the model makes predictions on a small batch of examples, measures its error and adjusts its settings once. Training runs consist of thousands to millions of steps., weight decayweight decay, n.: A training rule that shrinks every weight slightly at each step, like a small tax on large numbers. It keeps models simpler and usually helps them work better on new data., AlgoPerfAlgoPerf, prop. n.: A public contest that measures which methods for training neural networks reach a set level of accuracy fastest on identical hardware, across several kinds of tasks, without being hand-tuned for each one., IsoFLOP analysisIsoFLOP analysis, n.: A way of finding the best model size for a given amount of computing: train several models of different sizes at exactly the same computing cost and see which one ends up best., noise schedulenoise schedule, n.: The plan for how much static is mixed into training examples at each step of a diffusion model, from almost none to total noise; its shape affects quality and speed., schedule-free optimizationschedule-free optimization, n.: A way of training that needs no plan for slowing learning down over time, so training can be stopped or extended at any point and still produce a well-settled model., warmup-stable-decay schedulewarmup-stable-decay schedule, n.: A plan for how fast a model learns during training: speed up briefly, hold steady for most of the run, then slow down sharply at the end — which lets labs stop or extend training without starting over.