Skip to content
Last Dictionary
Plane
Deep LearningLanguage Models & TransformersVision, Speech & Perception

SiLU

abbreviationAlso: sigmoid linear unit, sigmoid-weighted linear unit, Swish, swish function

In plain wordsA smooth activation rule that multiplies each value by a soft on/off switch computed from the value itself. It is close to ReLU but curved, and is used in many modern models.

  1. The activation functionactivation function, n.: The simple rule each artificial neuron applies to its total input to decide its output, such as 'keep positive values, turn negatives into zero'. Without it, a deep network would be no smarter than a single layer. SiLU(x) = x·σ(x), where σ is the logistic sigmoid functionsigmoid function, n.: An S-shaped curve that squeezes any number into a value between 0 and 1, so the result can be read as a probability or as how far a switch is turned on., so the input gates itself. Like GELUGELU, abbr.: A smooth version of the ReLU rule, used in models such as BERT and GPT: it keeps large positive values, removes large negative ones, and blends softly in between., to which it is numerically close, it is smooth, unbounded above, bounded below (minimum ≈ −0.28 near x ≈ −1.28) and non-monotonic for negative inputs. The Swish form x·σ(βx) adds a fixed or trainable β, interpolating between a scaled linear function (β → 0) and ReLUReLU, abbr.: The most common rule a neuron uses to produce its output: keep the number if it is positive, replace it with zero if it is negative. Simple, fast and surprisingly effective. (β → ∞). SiLU is used in vision networks such as EfficientNet and recent YOLO models and, as the gate of SwiGLUSwiGLU, n.: A design for the feed-forward layers inside most modern language models: one calculation produces values, another produces a smooth on/off switch, and the two are multiplied together., in the feed-forward layers of PaLM, LLaMA and most later open-weight language modelslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code..

    The feed-forward block uses SiLU on the gate projection only.

SiLU⁡(x)=x σ(x)=x1+e−x\operatorname{SiLU}(x) = x\,\sigma(x) = \frac{x}{1 + e^{-x}}

Origin

First proposed by Dan Hendrycks and Kevin Gimpel (2016) alongside GELU, then independently by Stefan Elfwing, Eiji Uchibe and Kenji Doya (February 2017), who named it the sigmoid-weighted linear unit and used it in reinforcement learning. Prajit Ramachandran, Barret Zoph and Quoc Le found it again by automated search in October 2017 and called it Swish; PyTorch adopted the name SiLU.

Usage note

The two names usually denote the same function with β = 1. 'Swish' spread first through vision work; 'SiLU' credits the earlier papers.

Related

See also
  • GELUGELU, abbr.: A smooth version of the ReLU rule, used in models such as BERT and GPT: it keeps large positive values, removes large negative ones, and blends softly in between.
  • SwiGLUSwiGLU, n.: A design for the feed-forward layers inside most modern language models: one calculation produces values, another produces a smooth on/off switch, and the two are multiplied together.
  • sigmoid functionsigmoid function, n.: An S-shaped curve that squeezes any number into a value between 0 and 1, so the result can be read as a probability or as how far a switch is turned on.
  • ReLUReLU, abbr.: The most common rule a neuron uses to produce its output: keep the number if it is positive, replace it with zero if it is negative. Simple, fast and surprisingly effective.
  • activation functionactivation function, n.: The simple rule each artificial neuron applies to its total input to decide its output, such as 'keep positive values, turn negatives into zero'. Without it, a deep network would be no smarter than a single layer.
  • gated linear unitgated linear unit, n.: A layer design in which one half of the computation acts as a set of dials, each between off and on, deciding how much of the other half passes through.

Sources

  1. Elfwing, S., Uchibe, E. & Doya, K. (2017). Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning. arXiv:1702.03118. arxiv.org/abs/1702.03118
  2. Ramachandran, P., Zoph, B. & Le, Q. V. (2017). Searching for Activation Functions. arXiv:1710.05941. arxiv.org/abs/1710.05941

Mentioned in

activation functionactivation function, n.: The simple rule each artificial neuron applies to its total input to decide its output, such as 'keep positive values, turn negatives into zero'. Without it, a deep network would be no smarter than a single layer., GELUGELU, abbr.: A smooth version of the ReLU rule, used in models such as BERT and GPT: it keeps large positive values, removes large negative ones, and blends softly in between., ReLUReLU, abbr.: The most common rule a neuron uses to produce its output: keep the number if it is positive, replace it with zero if it is negative. Simple, fast and surprisingly effective., dying ReLUdying ReLU, n.: A training failure in which some of a network's units get stuck permanently switched off, always outputting zero, so they stop learning and become dead weight., EfficientNetEfficientNet, prop. n.: A family of image-recognition networks from 2019 that grows a small base design in a balanced way, deeper, wider and sharper together, to get more accuracy for each unit of computing., gated linear unitgated linear unit, n.: A layer design in which one half of the computation acts as a set of dials, each between off and on, deciding how much of the other half passes through., SwiGLUSwiGLU, n.: A design for the feed-forward layers inside most modern language models: one calculation produces values, another produces a smooth on/off switch, and the two are multiplied together.