dying ReLU
In plain wordsA training failure in which some of a network's units get stuck permanently switched off, always outputting zero, so they stop learning and become dead weight.
A failure mode of networks using the ReLUReLU, abbr.: The most common rule a neuron uses to produce its output: keep the number if it is positive, replace it with zero if it is negative. Simple, fast and surprisingly effective. activation, which outputs zero for negative inputs and has zero gradient there: if a unit's pre-activation becomes negative for every input — typically after a large update drives its bias strongly negative, or from unlucky initializationweight initialization, n.: Choosing the random starting values of a neural network's internal numbers before training. Good starting values keep signals from fading away or blowing up as they pass through many layers. — it outputs zero everywhere, receives no gradient, and cannot recover. A network can lose a large fraction of its units this way, wasting capacity; for narrow, deep networks the whole network can even be 'born dead', computing a constant function, with a probability that grows with depth. Contributing causes include learning rates that are too high and poorly scaled inputs. Remedies include lower learning rates or warmuplearning rate warmup, n.: Starting training with tiny learning steps and gradually increasing them over the first stretch of training, so the model does not lurch wildly before it has found its footing., careful initialization, normalization layers, and activations with nonzero gradient for negative inputs, such as leaky ReLUleaky ReLU, n.: A variant of the common ReLU rule that lets a small fraction of each negative value through instead of zeroing it, so neurons never go completely silent., ELU, GELUGELU, abbr.: A smooth version of the ReLU rule, used in models such as BERT and GPT: it keeps large positive values, removes large negative ones, and blends softly in between. and SiLUSiLU, abbr.: A smooth activation rule that multiplies each value by a soft on/off switch computed from the value itself. It is close to ReLU but curved, and is used in many modern models. — one reason for their adoption in modern architectures.
A third of the units in layer 4 were dead ReLUs after the learning-rate spike.
Origin
Recognized soon after ReLUs became popular around 2010–2012; Andrew Maas, Awni Hannun and Andrew Ng proposed the leaky ReLU in 2013 partly to keep gradients flowing through inactive units, and the phrase 'dying ReLU' was common in course notes and forums by about 2015. Lu Lu and colleagues analyzed the problem formally in 2019.
Related
- ReLUReLU, abbr.: The most common rule a neuron uses to produce its output: keep the number if it is positive, replace it with zero if it is negative. Simple, fast and surprisingly effective.
- leaky ReLUleaky ReLU, n.: A variant of the common ReLU rule that lets a small fraction of each negative value through instead of zeroing it, so neurons never go completely silent.
- GELUGELU, abbr.: A smooth version of the ReLU rule, used in models such as BERT and GPT: it keeps large positive values, removes large negative ones, and blends softly in between.
- activation functionactivation function, n.: The simple rule each artificial neuron applies to its total input to decide its output, such as 'keep positive values, turn negatives into zero'. Without it, a deep network would be no smarter than a single layer.
- weight initializationweight initialization, n.: Choosing the random starting values of a neural network's internal numbers before training. Good starting values keep signals from fading away or blowing up as they pass through many layers.
- vanishing gradient problemvanishing gradient problem, n.: A training difficulty in deep or recurrent networks: the learning signal shrinks as it travels back through many layers or time steps, so the early parts of the network barely learn.
Sources
- Lu, L., Shin, Y., Su, Y. & Karniadakis, G. E. (2020). Dying ReLU and Initialization: Theory and Numerical Examples. Communications in Computational Physics 28(5). arxiv.org/abs/1903.06733
- Maas, A. L., Hannun, A. Y. & Ng, A. Y. (2013). Rectifier Nonlinearities Improve Neural Network Acoustic Models. ICML Workshop on Deep Learning for Audio, Speech and Language Processing.
Mentioned in
ReLUReLU, abbr.: The most common rule a neuron uses to produce its output: keep the number if it is positive, replace it with zero if it is negative. Simple, fast and surprisingly effective., leaky ReLUleaky ReLU, n.: A variant of the common ReLU rule that lets a small fraction of each negative value through instead of zeroing it, so neurons never go completely silent., loss of plasticityloss of plasticity, n.: The gradual loss of a neural network's ability to learn new things after long training on changing tasks, as parts of the network go inactive or become stuck.