exploding gradient problem
In plain wordsA training failure in which the learning signal grows out of control as it passes back through a network, so the weights jump wildly and training breaks down.
The tendency of gradients computed by backpropagationbackpropagation, n.: The method neural networks use to learn from mistakes: it works backward from the error to figure out how much each internal connection contributed, so every connection can be adjusted in the right direction. to grow exponentially as they pass backward through many layers or time steps, when the Jacobian factors in the chain typically have norms above 1, as with large recurrent weights. Huge gradients produce enormous parameter updates that overshoot, causing sudden loss increases, numerical overflow (NaN or infinite values) or divergence. In recurrent networks, Razvan Pascanu and colleagues linked explosions to steep, wall-like regions of the loss surface, often called cliffs. The standard remedy is gradient clippinggradient clipping, n.: A safety limit on how big a single training update can be. If the computed adjustment is unusually large, it is scaled down so that one bad batch cannot wreck the model., which rescales the gradient when its norm exceeds a threshold; others include careful weight initializationweight initialization, n.: Choosing the random starting values of a neural network's internal numbers before training. Good starting values keep signals from fading away or blowing up as they pass through many layers., normalizationnormalization layer, n.: A layer that keeps the numbers passing through a neural network within a steady range by shifting and resizing them, so that deep networks learn faster and are less likely to break down during training., smaller learning rateslearning rate, n.: How big a step a model takes each time it adjusts its settings during training. Too large and learning goes haywire; too small and it crawls. with warmuplearning rate warmup, n.: Starting training with tiny learning steps and gradually increasing them over the first stretch of training, so the model does not lurch wildly before it has found its footing., and gated architectures. Sporadic loss spikesloss spike, n.: A sudden jump in a model's error partway through training. Sometimes the model recovers on its own; sometimes the run falls apart and engineers must rewind to an earlier saved copy. in large-model training are a related concern (training instabilitytraining instability, n.: The tendency of very large training runs to go wrong — the error suddenly jumping or blowing up — which can waste weeks of expensive computing unless engineers prevent or repair it.).
The loss went to NaN at step 3,000; clipping the gradient norm at 1.0 fixed the explosion.
Origin
Analyzed together with vanishing gradients in Sepp Hochreiter's 1991 diploma thesis, which showed that errors backpropagated through a recurrent network tend either to grow or to decay exponentially, and by Yoshua Bengio, Patrice Simard and Paolo Frasconi (1994). Razvan Pascanu, Tomas Mikolov and Bengio's 'On the Difficulty of Training Recurrent Neural Networks' (arXiv, November 2012; ICML 2013) analyzed it in detail and popularized gradient-norm clipping as the fix.
Related
- vanishing gradient problemvanishing gradient problem, n.: A training difficulty in deep or recurrent networks: the learning signal shrinks as it travels back through many layers or time steps, so the early parts of the network barely learn.
- gradient clippinggradient clipping, n.: A safety limit on how big a single training update can be. If the computed adjustment is unusually large, it is scaled down so that one bad batch cannot wreck the model.
- loss spikeloss spike, n.: A sudden jump in a model's error partway through training. Sometimes the model recovers on its own; sometimes the run falls apart and engineers must rewind to an earlier saved copy.
- training instabilitytraining instability, n.: The tendency of very large training runs to go wrong — the error suddenly jumping or blowing up — which can waste weeks of expensive computing unless engineers prevent or repair it.
- recurrent neural networkrecurrent neural network, n.: A neural network that reads a sequence, such as a sentence or an audio clip, one piece at a time, carrying a running summary forward so that what came earlier can shape how later pieces are understood.
- weight initializationweight initialization, n.: Choosing the random starting values of a neural network's internal numbers before training. Good starting values keep signals from fading away or blowing up as they pass through many layers.
- vanishing gradient problemvanishing gradient problem, n.: A training difficulty in deep or recurrent networks: the learning signal shrinks as it travels back through many layers or time steps, so the early parts of the network barely learn.
Sources
- Pascanu, R., Mikolov, T. & Bengio, Y. (2012). On the Difficulty of Training Recurrent Neural Networks. arXiv:1211.5063; ICML 2013. arxiv.org/abs/1211.5063
Mentioned in
backpropagation through timebackpropagation through time, n.: The way recurrent networks are trained: the network is unrolled into a long chain, one copy per time step, and errors are traced back through every step to adjust the shared weights., chain rulechain rule, n.: A rule for working out how a change ripples through a chain of steps by combining the effect at each step. It is how AI training traces each error back through every layer., eigenvectoreigenvector, n.: A special direction for a transformation: anything pointing that way stays on its line, only longer, shorter or reversed; the amount of stretch is the eigenvalue. Together they reveal the main axes along which data or a system varies., gradientgradient, n.: A list of slopes showing which way, and how steeply, a quantity rises as each of its inputs changes. Training nudges an AI's settings in the opposite direction, downhill., gradient clippinggradient clipping, n.: A safety limit on how big a single training update can be. If the computed adjustment is unusually large, it is scaled down so that one bad batch cannot wreck the model., recurrent neural networkrecurrent neural network, n.: A neural network that reads a sequence, such as a sentence or an audio clip, one piece at a time, carrying a running summary forward so that what came earlier can shape how later pieces are understood., training instabilitytraining instability, n.: The tendency of very large training runs to go wrong — the error suddenly jumping or blowing up — which can waste weeks of expensive computing unless engineers prevent or repair it., vanishing gradient problemvanishing gradient problem, n.: A training difficulty in deep or recurrent networks: the learning signal shrinks as it travels back through many layers or time steps, so the early parts of the network barely learn., weight initializationweight initialization, n.: Choosing the random starting values of a neural network's internal numbers before training. Good starting values keep signals from fading away or blowing up as they pass through many layers., Tomas MikolovTomas Mikolov, prop. n.: A Czech computer scientist who created word2vec, a fast way to turn words into lists of numbers that capture their meaning, and pioneered language models built from recurrent networks.