vanishing gradient problem
In plain wordsA training difficulty in deep or recurrent networks: the learning signal shrinks as it travels back through many layers or time steps, so the early parts of the network barely learn.
The tendency of gradients computed by backpropagationbackpropagation, n.: The method neural networks use to learn from mistakes: it works backward from the error to figure out how much each internal connection contributed, so every connection can be adjusted in the right direction. to shrink exponentially as they pass backward through many layers or time steps, so that early layers of a deep network, or long-range dependencies in a recurrent neural networkrecurrent neural network, n.: A neural network that reads a sequence, such as a sentence or an audio clip, one piece at a time, carrying a running summary forward so that what came earlier can shape how later pieces are understood., receive almost no learning signal. The gradient at an early layer is a product of many Jacobian factors; when their norms are typically below 1, as with saturating sigmoidsigmoid function, n.: An S-shaped curve that squeezes any number into a value between 0 and 1, so the result can be read as a probability or as how far a switch is turned on. or tanhtanh, n.: An S-shaped curve, like the sigmoid but centered on zero, that squeezes any number into a value between −1 and 1. It was the standard neuron rule before ReLU took over. units (the sigmoid's derivative never exceeds ¼) or recurrent weight matrices with spectral radius below 1, the product decays geometrically. Remedies that made deep learning practical include gated, additive state paths (LSTMlong short-term memory, n.: A type of recurrent network with a separate memory lane and learned 'gates' that decide what to remember, forget and output, letting it keep track of information over long sequences., GRUgated recurrent unit, n.: A simpler cousin of the LSTM: a recurrent network unit with two learned gates that decide how much of its memory to keep and how much to overwrite at each step.), residual connectionsresidual connection, n.: A shortcut in a neural network that adds a block's input straight to its output, so each block only has to learn a correction. It is what makes networks hundreds of layers deep trainable., non-saturating activations such as ReLUReLU, abbr.: The most common rule a neuron uses to produce its output: keep the number if it is positive, replace it with zero if it is negative. Simple, fast and surprisingly effective., careful weight initializationweight initialization, n.: Choosing the random starting values of a neural network's internal numbers before training. Good starting values keep signals from fading away or blowing up as they pass through many layers., normalization layersnormalization layer, n.: A layer that keeps the numbers passing through a neural network within a steady range by shifting and resizing them, so that deep networks learn faster and are less likely to break down during training. and, historically, unsupervised layer-wise pretraining.
With sigmoid activations the first layers' gradients were a million times smaller than the last layer's: classic vanishing gradients.
Origin
First formally analyzed by Sepp Hochreiter in his 1991 diploma thesis at the Technical University of Munich, supervised by Jürgen Schmidhuber, and independently by Yoshua Bengio, Patrice Simard and Paolo Frasconi in 'Learning Long-Term Dependencies with Gradient Descent is Difficult' (1994). Razvan Pascanu, Tomas Mikolov and Bengio's 2012 analysis treated vanishing and exploding gradients together.
Related
- exploding gradient problemexploding gradient problem, n.: A training failure in which the learning signal grows out of control as it passes back through a network, so the weights jump wildly and training breaks down.
- backpropagationbackpropagation, n.: The method neural networks use to learn from mistakes: it works backward from the error to figure out how much each internal connection contributed, so every connection can be adjusted in the right direction.
- long short-term memorylong short-term memory, n.: A type of recurrent network with a separate memory lane and learned 'gates' that decide what to remember, forget and output, letting it keep track of information over long sequences.
- residual connectionresidual connection, n.: A shortcut in a neural network that adds a block's input straight to its output, so each block only has to learn a correction. It is what makes networks hundreds of layers deep trainable.
- ReLUReLU, abbr.: The most common rule a neuron uses to produce its output: keep the number if it is positive, replace it with zero if it is negative. Simple, fast and surprisingly effective.
- weight initializationweight initialization, n.: Choosing the random starting values of a neural network's internal numbers before training. Good starting values keep signals from fading away or blowing up as they pass through many layers.
- backpropagation through timebackpropagation through time, n.: The way recurrent networks are trained: the network is unrolled into a long chain, one copy per time step, and errors are traced back through every step to adjust the shared weights.
- exploding gradient problemexploding gradient problem, n.: A training failure in which the learning signal grows out of control as it passes back through a network, so the weights jump wildly and training breaks down.
Sources
- Bengio, Y., Simard, P. & Frasconi, P. (1994). Learning Long-Term Dependencies with Gradient Descent is Difficult. IEEE Transactions on Neural Networks, 5(2), 157–166.
- Pascanu, R., Mikolov, T. & Bengio, Y. (2012). On the Difficulty of Training Recurrent Neural Networks. arXiv:1211.5063; ICML 2013. arxiv.org/abs/1211.5063
Mentioned in
backpropagationbackpropagation, n.: The method neural networks use to learn from mistakes: it works backward from the error to figure out how much each internal connection contributed, so every connection can be adjusted in the right direction., activation functionactivation function, n.: The simple rule each artificial neuron applies to its total input to decide its output, such as 'keep positive values, turn negatives into zero'. Without it, a deep network would be no smarter than a single layer., backpropagation through timebackpropagation through time, n.: The way recurrent networks are trained: the network is unrolled into a long chain, one copy per time step, and errors are traced back through every step to adjust the shared weights., backward passbackward pass, n.: The step in training where the network works backward from its error, layer by layer, to calculate how each internal setting should change., chain rulechain rule, n.: A rule for working out how a change ripples through a chain of steps by combining the effect at each step. It is how AI training traces each error back through every layer., eigenvectoreigenvector, n.: A special direction for a transformation: anything pointing that way stays on its line, only longer, shorter or reversed; the amount of stretch is the eigenvalue. Together they reveal the main axes along which data or a system varies., exploding gradient problemexploding gradient problem, n.: A training failure in which the learning signal grows out of control as it passes back through a network, so the weights jump wildly and training breaks down., gated recurrent unitgated recurrent unit, n.: A simpler cousin of the LSTM: a recurrent network unit with two learned gates that decide how much of its memory to keep and how much to overwrite at each step., gradientgradient, n.: A list of slopes showing which way, and how steeply, a quantity rises as each of its inputs changes. Training nudges an AI's settings in the opposite direction, downhill., Jürgen SchmidhuberJürgen Schmidhuber, prop. n.: A German computer scientist who co-invented the LSTM, a neural network with memory that powered much of speech recognition and translation in the 2010s, and who argues that early pioneers go under-credited., long short-term memorylong short-term memory, n.: A type of recurrent network with a separate memory lane and learned 'gates' that decide what to remember, forget and output, letting it keep track of information over long sequences., recurrent neural networkrecurrent neural network, n.: A neural network that reads a sequence, such as a sentence or an audio clip, one piece at a time, carrying a running summary forward so that what came earlier can shape how later pieces are understood., ReLUReLU, abbr.: The most common rule a neuron uses to produce its output: keep the number if it is positive, replace it with zero if it is negative. Simple, fast and surprisingly effective., residual connectionresidual connection, n.: A shortcut in a neural network that adds a block's input straight to its output, so each block only has to learn a correction. It is what makes networks hundreds of layers deep trainable., sigmoid functionsigmoid function, n.: An S-shaped curve that squeezes any number into a value between 0 and 1, so the result can be read as a probability or as how far a switch is turned on., tanhtanh, n.: An S-shaped curve, like the sigmoid but centered on zero, that squeezes any number into a value between −1 and 1. It was the standard neuron rule before ReLU took over., weight initializationweight initialization, n.: Choosing the random starting values of a neural network's internal numbers before training. Good starting values keep signals from fading away or blowing up as they pass through many layers., Yoshua BengioYoshua Bengio, prop. n.: A Canadian computer scientist who helped found deep learning, including ideas behind today's language models, and who has become a leading voice on the risks of advanced AI., dying ReLUdying ReLU, n.: A training failure in which some of a network's units get stuck permanently switched off, always outputting zero, so they stop learning and become dead weight., highway networkhighway network, n.: An early design for very deep networks in which learned gates let information either pass through a layer unchanged or be transformed, making networks hundreds of layers deep trainable., Sepp HochreiterSepp Hochreiter, prop. n.: A German computer scientist who explained why early neural networks struggled to learn long sequences and, with Jürgen Schmidhuber, invented the LSTM network that solved the problem.