Skip to content
Last Dictionary
Plane
Deep Learning

leaky ReLU

nounAlso: LReLU, leaky rectified linear unit, parametric ReLU, PReLU

In plain wordsA variant of the common ReLU rule that lets a small fraction of each negative value through instead of zeroing it, so neurons never go completely silent.

  1. A variant of ReLUReLU, abbr.: The most common rule a neuron uses to produce its output: keep the number if it is positive, replace it with zero if it is negative. Simple, fast and surprisingly effective. that gives negative inputs a small nonzero slope α: f(x) = x for x > 0 and αx otherwise, with α typically 0.01 and sometimes 0.1–0.3. Because the gradient is never exactly zero, units cannot become permanently inactive (dying ReLUdying ReLU, n.: A training failure in which some of a network's units get stuck permanently switched off, always outputting zero, so they stop learning and become dead weight.). In the parametric ReLU (PReLU), α is learned during training, per layer or per channel. Leaky ReLU is common in GANgenerative adversarial network, n.: Two neural networks trained as rivals: one forges fake images, the other tries to spot the fakes. As both improve, the forger's output becomes realistic enough to fool people. discriminators and some detection networks. Related fixes for the negative side include the exponential linear unit (ELU), which saturates smoothly towards −α and pushes mean activations towards zero, and its self-normalizing scaled version, SELU.

    The discriminator uses leaky ReLU with a slope of 0.2 so that gradients still reach the generator.

LeakyReLU⁡(x)={xx>0αxx≤0\operatorname{LeakyReLU}(x) = \begin{cases} x & x > 0 \\ \alpha x & x \le 0 \end{cases}

Origin

Introduced by Andrew Maas, Awni Hannun and Andrew Ng in 'Rectifier Nonlinearities Improve Neural Network Acoustic Models', presented at an ICML 2013 workshop. Kaiming He and colleagues made the slope learnable (PReLU) in February 2015, reporting 4.94% top-5 ImageNet error, the first result below a reported human error of 5.1%. ELU followed from Djork-Arné Clevert and colleagues in November 2015.

Related

See also
  • ReLUReLU, abbr.: The most common rule a neuron uses to produce its output: keep the number if it is positive, replace it with zero if it is negative. Simple, fast and surprisingly effective.
  • dying ReLUdying ReLU, n.: A training failure in which some of a network's units get stuck permanently switched off, always outputting zero, so they stop learning and become dead weight.
  • activation functionactivation function, n.: The simple rule each artificial neuron applies to its total input to decide its output, such as 'keep positive values, turn negatives into zero'. Without it, a deep network would be no smarter than a single layer.
  • GELUGELU, abbr.: A smooth version of the ReLU rule, used in models such as BERT and GPT: it keeps large positive values, removes large negative ones, and blends softly in between.
  • generative adversarial networkgenerative adversarial network, n.: Two neural networks trained as rivals: one forges fake images, the other tries to spot the fakes. As both improve, the forger's output becomes realistic enough to fool people.

Sources

  1. He, K., Zhang, X., Ren, S. & Sun, J. (2015). Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. arXiv:1502.01852. arxiv.org/abs/1502.01852
  2. Clevert, D.-A., Unterthiner, T. & Hochreiter, S. (2015). Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs). arXiv:1511.07289. arxiv.org/abs/1511.07289
  3. Maas, A. L., Hannun, A. Y. & Ng, A. Y. (2013). Rectifier Nonlinearities Improve Neural Network Acoustic Models. ICML 2013 Workshop on Deep Learning for Audio, Speech and Language Processing.

Mentioned in

activation functionactivation function, n.: The simple rule each artificial neuron applies to its total input to decide its output, such as 'keep positive values, turn negatives into zero'. Without it, a deep network would be no smarter than a single layer., ReLUReLU, abbr.: The most common rule a neuron uses to produce its output: keep the number if it is positive, replace it with zero if it is negative. Simple, fast and surprisingly effective., dying ReLUdying ReLU, n.: A training failure in which some of a network's units get stuck permanently switched off, always outputting zero, so they stop learning and become dead weight.