Skip to content
Last Dictionary
Plane
Deep LearningLanguage Models & TransformersTraining & Post-training

layer normalization

nounAlso: LayerNorm, layer norm, LN, pre-norm, post-norm, pre-LN, post-LN

In plain wordsA layer that rescales the numbers inside a network, separately for each example, so they keep a steady average and spread. It, or a simpler variant, is a standard part of the models behind today's chatbots.

  1. A normalization layernormalization layer, n.: A layer that keeps the numbers passing through a neural network within a steady range by shifting and resizing them, so that deep networks learn faster and are less likely to break down during training. that normalizes the activations of a single example across its feature dimension, subtracting their mean and dividing by their standard deviation, then applies a learned per-feature gain and bias. Because its statistics do not depend on the batch, it behaves identically in training and inference, works with a batch of one and handles variable-length sequences, which made it the standard normalization for RNNsrecurrent neural network, n.: A neural network that reads a sequence, such as a sentence or an audio clip, one piece at a time, carrying a running summary forward so that what came earlier can shape how later pieces are understood. and transformerstransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it.. Placement matters: the original transformer used post-norm, normalizing after each residual addition, but most large models since GPT-2 use pre-norm, normalizing the input to each sublayer and leaving the residualresidual connection, n.: A shortcut in a neural network that adds a block's input straight to its output, so each block only has to learn a correction. It is what makes networks hundreds of layers deep trainable. path untouched, which trains more stably and depends less on learning rate warmuplearning rate warmup, n.: Starting training with tiny learning steps and gradually increasing them over the first stretch of training, so the model does not lurch wildly before it has found its footing.. Many recent models use the cheaper RMSNormRMSNorm, n.: A simpler, faster version of the layer that keeps a network's internal numbers at a steady scale. Most recent large language models use it. instead.

    Moving layer normalization to the start of each block let us drop the warmup schedule.

LN⁡(x)=γ⊙x−μσ2+ϵ+β\operatorname{LN}(\mathbf{x}) = \boldsymbol{\gamma} \odot \frac{\mathbf{x} - \mu}{\sqrt{\sigma^2 + \epsilon}} + \boldsymbol{\beta}

Origin

Introduced by Jimmy Lei Ba, Jamie Ryan Kiros and Geoffrey Hinton in 'Layer Normalization' (arXiv, July 2016) as a batch-independent alternative to batch normalization for recurrent networks. The transformer (2017) adopted it, and Ruibin Xiong and colleagues (2020) analyzed why pre-norm placement trains more stably than post-norm.

Usage note

Normalization-free transformers are a research topic: in 2025 Jiachen Zhu, Xinlei Chen, Kaiming He, Yann LeCun and Zhuang Liu replaced it with an elementwise, learnably scaled tanhtanh, n.: An S-shaped curve, like the sigmoid but centered on zero, that squeezes any number into a value between −1 and 1. It was the standard neuron rule before ReLU took over. ('Dynamic Tanh'), matching normalized models in their experiments.

Related

See also
  • batch normalizationbatch normalization, n.: A layer that rescales the numbers flowing through a network so that, across each batch of training examples, they keep a steady average and spread. It makes deep networks train faster and more reliably.
  • RMSNormRMSNorm, n.: A simpler, faster version of the layer that keeps a network's internal numbers at a steady scale. Most recent large language models use it.
  • normalization layernormalization layer, n.: A layer that keeps the numbers passing through a neural network within a steady range by shifting and resizing them, so that deep networks learn faster and are less likely to break down during training.
  • transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it.
  • residual connectionresidual connection, n.: A shortcut in a neural network that adds a block's input straight to its output, so each block only has to learn a correction. It is what makes networks hundreds of layers deep trainable.
  • residual streamresidual stream, n.: The running internal state that flows through a transformer model from layer to layer; each layer reads from it and adds its own contribution, like editors annotating a shared document.
  • learning rate warmuplearning rate warmup, n.: Starting training with tiny learning steps and gradually increasing them over the first stretch of training, so the model does not lurch wildly before it has found its footing.

Sources

  1. Ba, J. L., Kiros, J. R. & Hinton, G. E. (2016). Layer Normalization. arXiv:1607.06450. arxiv.org/abs/1607.06450
  2. Xiong, R. et al. (2020). On Layer Normalization in the Transformer Architecture. arXiv:2002.04745; ICML 2020. arxiv.org/abs/2002.04745

Mentioned in

transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it., batch normalizationbatch normalization, n.: A layer that rescales the numbers flowing through a network so that, across each batch of training examples, they keep a steady average and spread. It makes deep networks train faster and more reliably., feature scalingfeature scaling, n.: Putting input measurements on comparable scales, so that income in dollars and age in years, say, both span similar ranges and no feature dominates just because of its units., learning rate warmuplearning rate warmup, n.: Starting training with tiny learning steps and gradually increasing them over the first stretch of training, so the model does not lurch wildly before it has found its footing., normnorm, n.: A way of measuring the size, or length, of a list of numbers — for example the straight-line distance from the origin to the point the list describes., normalization layernormalization layer, n.: A layer that keeps the numbers passing through a neural network within a steady range by shifting and resizing them, so that deep networks learn faster and are less likely to break down during training., residual connectionresidual connection, n.: A shortcut in a neural network that adds a block's input straight to its output, so each block only has to learn a correction. It is what makes networks hundreds of layers deep trainable., residual streamresidual stream, n.: The running internal state that flows through a transformer model from layer to layer; each layer reads from it and adds its own contribution, like editors annotating a shared document., tanhtanh, n.: An S-shaped curve, like the sigmoid but centered on zero, that squeezes any number into a value between −1 and 1. It was the standard neuron rule before ReLU took over., transformer blocktransformer block, n.: The repeating building unit of a transformer. A model is a stack of dozens of identical blocks, each pairing an attention step that mixes information between words with a step that processes each word., variancevariance, n.: A number that grows the more widely something's possible values scatter away from its average, and is zero if it never varies. Its square root, the standard deviation, uses the same units as the values themselves., ConvNeXtConvNeXt, prop. n.: A 2022 image-recognition network showing that an older-style convolutional design, updated with lessons from transformers, can match vision transformers., direct logit attributiondirect logit attribution, n.: A quick way to measure how much each internal part of a language model directly pushed it toward a particular next word, by splitting the final score into separate contributions., kernel fusionkernel fusion, n.: Merging several small calculation steps into one, so a chip finishes them in a single pass instead of saving and reloading data in between, like cooking at the counter without returning ingredients to the fridge after each step., logit lenslogit lens, n.: A quick trick for peeking at what a language model is 'thinking' partway through: read its half-finished internal state as if it were the final answer, one layer at a time., QK-normQK-norm, n.: A small change to how a language model pays attention — rescaling two sets of internal numbers before comparing them — that prevents a common way large models' training goes haywire., RMSNormRMSNorm, n.: A simpler, faster version of the layer that keeps a network's internal numbers at a steady scale. Most recent large language models use it., sequence parallelismsequence parallelism, n.: Splitting the long string of words a model is working on into chunks handled by different chips, which saves memory and makes it possible to train on very long documents., SmoothQuantSmoothQuant, prop. n.: A method that lets large language models run entirely on 8-bit numbers by moving hard-to-round extreme values out of the data flowing through the model and into its fixed numbers, which cope better.