Skip to content
Last Dictionary
Plane
Deep LearningTraining & Post-training

cosine annealing

nounAlso: cosine schedule, cosine decay, cosine learning rate schedule, SGDR, warm restarts

In plain wordsA popular plan for shrinking a model's step size during training: steps start large, shrink slowly, then faster, then level off near zero, following the shape of half a wave.

  1. A learning rate schedulelearning rate schedule, n.: A plan for changing how big a model's learning steps are during training — usually starting cautiously, then taking big steps, then smaller and smaller ones as it settles in. that lowers the learning ratelearning rate, n.: How big a step a model takes each time it adjusts its settings during training. Too large and learning goes haywire; too small and it crawls. from a maximum η_max to a minimum η_min along half a cosine period over T steps. The rate stays near its peak early, falls fastest mid-training and flattens near the end. In the original form, SGDR ('stochastic gradient descent with warm restarts'), the schedule jumps back to η_max periodically, often with lengthening periods, and the snapshot at the end of each cycle can be ensembled; most practice now uses a single cycle preceded by learning rate warmuplearning rate warmup, n.: Starting training with tiny learning steps and gradually increasing them over the first stretch of training, so the model does not lurch wildly before it has found its footing.. Single-cycle cosine decay to about 10% of the peak was the default for large language modellarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. pretraining from about 2020 to 2023. Because T must be fixed in advance, a finished run cannot simply be extended, and stopping early leaves the rate too high — drawbacks that motivated the warmup-stable-decay schedulewarmup-stable-decay schedule, n.: A plan for how fast a model learns during training: speed up briefly, hold steady for most of the run, then slow down sharply at the end — which lets labs stop or extend training without starting over. and schedule-free optimizationschedule-free optimization, n.: A way of training that needs no plan for slowing learning down over time, so training can be stopped or extended at any point and still produce a well-settled model..

    We use linear warmup for 2,000 steps, then cosine decay to 10% of the peak learning rate.

ηt=ηmin⁡+12(ηmax⁡−ηmin⁡)(1+cos⁡πtT)\eta_t = \eta_{\min} + \tfrac{1}{2}\left(\eta_{\max} - \eta_{\min}\right)\left(1 + \cos\frac{\pi t}{T}\right)

Origin

Introduced by Ilya Loshchilov and Frank Hutter in 'SGDR: Stochastic Gradient Descent with Warm Restarts' (arXiv 2016; ICLR 2017). 'Annealing' borrows the metallurgical metaphor by way of simulated annealing. Single-cycle cosine decay was used for GPT-3 (2020), Chinchilla (2022) and LLaMA (2023).

Usage note

The Chinchilla authors reported that a cosine cycle much longer than the actual run degrades final loss, so T should match the true training length.

Related

See also
  • learning rate schedulelearning rate schedule, n.: A plan for changing how big a model's learning steps are during training — usually starting cautiously, then taking big steps, then smaller and smaller ones as it settles in.
  • learning rate warmuplearning rate warmup, n.: Starting training with tiny learning steps and gradually increasing them over the first stretch of training, so the model does not lurch wildly before it has found its footing.
  • warmup-stable-decay schedulewarmup-stable-decay schedule, n.: A plan for how fast a model learns during training: speed up briefly, hold steady for most of the run, then slow down sharply at the end — which lets labs stop or extend training without starting over.
  • schedule-free optimizationschedule-free optimization, n.: A way of training that needs no plan for slowing learning down over time, so training can be stopped or extended at any point and still produce a well-settled model.
  • simulated annealingsimulated annealing, n.: An optimization method inspired by slowly cooling metal: early on it sometimes accepts worse solutions, to escape dead ends, and becomes pickier as a 'temperature' setting gradually drops.
  • Chinchilla scaling lawsChinchilla scaling laws, n.: A 2022 DeepMind finding that, for a fixed amount of computing, language models had been made too big and trained on too little text; the best balance is roughly 20 word-pieces of text for every adjustable number inside the model.

Sources

  1. Loshchilov, I. & Hutter, F. (2017). SGDR: Stochastic Gradient Descent with Warm Restarts. ICLR 2017. arxiv.org/abs/1608.03983

Mentioned in

annealingannealing, n.: The last stretch of training, when the model's learning is slowed to a stop — often while it is fed a small helping of the best data — so it settles into its final, sharpest version., learning rate schedulelearning rate schedule, n.: A plan for changing how big a model's learning steps are during training — usually starting cautiously, then taking big steps, then smaller and smaller ones as it settles in., schedule-free optimizationschedule-free optimization, n.: A way of training that needs no plan for slowing learning down over time, so training can be stopped or extended at any point and still produce a well-settled model., warmup-stable-decay schedulewarmup-stable-decay schedule, n.: A plan for how fast a model learns during training: speed up briefly, hold steady for most of the run, then slow down sharply at the end — which lets labs stop or extend training without starting over.