GELU
In plain wordsA smooth version of the ReLU rule, used in models such as BERT and GPT: it keeps large positive values, removes large negative ones, and blends softly in between.
The activation functionactivation function, n.: The simple rule each artificial neuron applies to its total input to decide its output, such as 'keep positive values, turn negatives into zero'. Without it, a deep network would be no smarter than a single layer. GELU(x) = x·Φ(x), where Φ is the cumulative distribution function of the standard normal distributionnormal distribution, n.: The familiar bell-shaped curve: most values cluster near the average, and values become rapidly rarer the farther they lie from it. Measurement errors and adult heights roughly follow it.. It weights each input by the probability that a standard Gaussian variable falls below it, so it behaves like the identity for large positive x and like zero for large negative x, and is smooth and slightly non-monotonic in between, with a minimum of about −0.17 near x ≈ −0.75. Its authors motivated it as the expected output of a stochastic regularizer that randomly zeroes inputs, combining ideas from ReLUReLU, abbr.: The most common rule a neuron uses to produce its output: keep the number if it is positive, replace it with zero if it is negative. Simple, fast and surprisingly effective. and dropoutdropout, n.: A training trick that randomly switches off a fraction of a network's neurons at each step, so the network cannot lean too heavily on any one of them and learns sturdier patterns.. Common fast approximations are 0.5x(1 + tanh[√(2/π)(x + 0.044715x³)]) and x·σ(1.702x). GELU became the default nonlinearity of BERTBERT, prop. n.: An influential 2018 Google language model that learned to understand text by filling in hidden words. It made search engines and many language tools much better at grasping meaning in context., the GPTGPT, abbr.: Short for 'generative pre-trained transformer', the family of OpenAI models behind ChatGPT. The letters are also used loosely for any similar chatbot model. series and vision transformersvision transformer, n.: A way of applying the transformer, the design behind chatbots, to pictures: the image is cut into small square tiles that the model reads like words in a sentence., and is the gate in the GEGLU variant of gated linear unitsgated linear unit, n.: A layer design in which one half of the computation acts as a set of dials, each between off and on, deciding how much of the other half passes through..
The checkpoint was trained with the tanh approximation of GELU, so use the same one at inference.
Origin
Introduced by Dan Hendrycks and Kevin Gimpel in a June 2016 arXiv paper, which also proposed the sigmoid-weighted function later known as SiLU. Adoption in OpenAI's GPT (2018) and Google's BERT (2018) made it the standard transformer activation until gated variants such as SwiGLU (proposed in 2020) displaced it in many large language models from 2022.
Related
- activation functionactivation function, n.: The simple rule each artificial neuron applies to its total input to decide its output, such as 'keep positive values, turn negatives into zero'. Without it, a deep network would be no smarter than a single layer.
- ReLUReLU, abbr.: The most common rule a neuron uses to produce its output: keep the number if it is positive, replace it with zero if it is negative. Simple, fast and surprisingly effective.
- SiLUSiLU, abbr.: A smooth activation rule that multiplies each value by a soft on/off switch computed from the value itself. It is close to ReLU but curved, and is used in many modern models.
- SwiGLUSwiGLU, n.: A design for the feed-forward layers inside most modern language models: one calculation produces values, another produces a smooth on/off switch, and the two are multiplied together.
- transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it.
- BERTBERT, prop. n.: An influential 2018 Google language model that learned to understand text by filling in hidden words. It made search engines and many language tools much better at grasping meaning in context.
Sources
- Hendrycks, D. & Gimpel, K. (2016). Gaussian Error Linear Units (GELUs). arXiv:1606.08415. arxiv.org/abs/1606.08415
Mentioned in
activation functionactivation function, n.: The simple rule each artificial neuron applies to its total input to decide its output, such as 'keep positive values, turn negatives into zero'. Without it, a deep network would be no smarter than a single layer., Dan HendrycksDan Hendrycks, prop. n.: An American AI-safety researcher who created widely used AI tests such as MMLU and Humanity's Last Exam, and who runs the Center for AI Safety., feed-forward networkfeed-forward network, n.: The part of each transformer layer that works on every word separately, after attention has mixed in context. It holds most of a model's adjustable numbers and is thought to store much of its knowledge., multilayer perceptronmultilayer perceptron, n.: A neural network made of several layers of artificial neurons, each connected to every neuron in the next layer. It is the simplest kind of deep network and a component of almost every modern model., ReLUReLU, abbr.: The most common rule a neuron uses to produce its output: keep the number if it is positive, replace it with zero if it is negative. Simple, fast and surprisingly effective., SiLUSiLU, abbr.: A smooth activation rule that multiplies each value by a soft on/off switch computed from the value itself. It is close to ReLU but curved, and is used in many modern models., ConvNeXtConvNeXt, prop. n.: A 2022 image-recognition network showing that an older-style convolutional design, updated with lessons from transformers, can match vision transformers., dying ReLUdying ReLU, n.: A training failure in which some of a network's units get stuck permanently switched off, always outputting zero, so they stop learning and become dead weight., gated linear unitgated linear unit, n.: A layer design in which one half of the computation acts as a set of dials, each between off and on, deciding how much of the other half passes through., leaky ReLUleaky ReLU, n.: A variant of the common ReLU rule that lets a small fraction of each negative value through instead of zeroing it, so neurons never go completely silent., SwiGLUSwiGLU, n.: A design for the feed-forward layers inside most modern language models: one calculation produces values, another produces a smooth on/off switch, and the two are multiplied together.