Skip to content
Last Dictionary
Plane
Deep LearningTraining & Post-training

gradient clipping

nounAlso: gradient norm clipping, clip_grad_norm, value clipping

In plain wordsA safety limit on how big a single training update can be. If the computed adjustment is unusually large, it is scaled down so that one bad batch cannot wreck the model.

  1. A safeguard that limits the size of gradients before the optimizeroptimizer, n.: The rule a training program uses to turn measured errors into actual changes to a model's internal numbers, deciding the direction and size of each adjustment. step. In norm clipping, the usual form, the gradient vector across all parameters is rescaled to have norm c whenever its norm exceeds that threshold (1.0 is common for language models), preserving its direction; value clipping instead caps each component separately, which can change the direction. Clipping keeps outlier batches and sharp cliffs in the loss landscapeloss landscape, n.: A way of picturing training as a journey across hilly terrain, where position stands for the model's settings and height for its error. Training searches for low valleys. from producing huge updates — the exploding gradient problemexploding gradient problem, n.: A training failure in which the learning signal grows out of control as it passes back through a network, so the weights jump wildly and training breaks down. that first plagued recurrent networksrecurrent neural network, n.: A neural network that reads a sequence, such as a sentence or an audio clip, one piece at a time, carrying a running summary forward so that what came earlier can shape how later pieces are understood. — and is standard in transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it. pretraining, where the fraction of clipped steps is watched as an early warning of training instabilitytraining instability, n.: The tendency of very large training runs to go wrong — the error suddenly jumping or blowing up — which can waste weeks of expensive computing unless engineers prevent or repair it.. Clipping each example's gradient separately is the core of DP-SGDDP-SGD, abbr.: A way of training a neural network so that it cannot learn too much about any single person in the data: each example's influence is capped, and random noise is added at every step., where it bounds any one example's influence for differential privacydifferential privacy, n.: A mathematical promise that the results of analyzing a dataset — or a model trained on it — would look almost the same whether or not any one person's data was included, so little can be learned about that person.. Related safeguards act elsewhere: AdafactorAdafactor, prop. n.: A memory-saving cousin of the popular Adam training method. Instead of keeping extra statistics for every single weight, it keeps summaries for each row and column of a weight table. clips its updates, and the QK-clip used with MuonMuon, prop. n.: A newer training method for neural networks that tidies each update so it pushes evenly in all directions within a weight table. Several groups report that it trains language models with less computing than the standard method. in Kimi K2 rescales query and key weights.

    Clip the global gradient norm at 1.0 and log how often clipping fires.

g←g⋅min⁡ ⁣(1,c∥g∥)g \leftarrow g \cdot \min\!\left(1, \frac{c}{\lVert g \rVert}\right)

Origin

Tomas Mikolov clipped gradient values when training recurrent language models (PhD thesis, Brno University of Technology, 2012). Razvan Pascanu, Mikolov and Yoshua Bengio analyzed exploding gradients and proposed rescaling by the gradient norm in 'On the difficulty of training recurrent neural networks' (arXiv 2012; ICML 2013), the form now standard.

Related

See also
  • exploding gradient problemexploding gradient problem, n.: A training failure in which the learning signal grows out of control as it passes back through a network, so the weights jump wildly and training breaks down.
  • training instabilitytraining instability, n.: The tendency of very large training runs to go wrong — the error suddenly jumping or blowing up — which can waste weeks of expensive computing unless engineers prevent or repair it.
  • loss spikeloss spike, n.: A sudden jump in a model's error partway through training. Sometimes the model recovers on its own; sometimes the run falls apart and engineers must rewind to an earlier saved copy.
  • DP-SGDDP-SGD, abbr.: A way of training a neural network so that it cannot learn too much about any single person in the data: each example's influence is capped, and random noise is added at every step.
  • optimizeroptimizer, n.: The rule a training program uses to turn measured errors into actual changes to a model's internal numbers, deciding the direction and size of each adjustment.
  • recurrent neural networkrecurrent neural network, n.: A neural network that reads a sequence, such as a sentence or an audio clip, one piece at a time, carrying a running summary forward so that what came earlier can shape how later pieces are understood.

Sources

  1. Pascanu, R., Mikolov, T. & Bengio, Y. (2013). On the difficulty of training recurrent neural networks. ICML 2013. arxiv.org/abs/1211.5063

Mentioned in

AdamWAdamW, prop. n.: A corrected version of the popular Adam training method that applies its steady pull toward smaller weights separately from the main update. It is the standard choice for training large language models., exploding gradient problemexploding gradient problem, n.: A training failure in which the learning signal grows out of control as it passes back through a network, so the weights jump wildly and training breaks down., normnorm, n.: A way of measuring the size, or length, of a list of numbers — for example the straight-line distance from the origin to the point the list describes., numerical stabilitynumerical stability, n.: Whether a computer calculation stays accurate despite the tiny rounding errors in every step, instead of blowing up into nonsense such as infinities., training instabilitytraining instability, n.: The tendency of very large training runs to go wrong — the error suddenly jumping or blowing up — which can waste weeks of expensive computing unless engineers prevent or repair it., training looptraining loop, n.: The repeating cycle at the heart of training software: grab some examples, let the model guess, measure the error, adjust the model, and do it again — often millions of times., loss spikeloss spike, n.: A sudden jump in a model's error partway through training. Sometimes the model recovers on its own; sometimes the run falls apart and engineers must rewind to an earlier saved copy., Tomas MikolovTomas Mikolov, prop. n.: A Czech computer scientist who created word2vec, a fast way to turn words into lists of numbers that capture their meaning, and pioneered language models built from recurrent networks.