normalization layer
In plain wordsA layer that keeps the numbers passing through a neural network within a steady range by shifting and resizing them, so that deep networks learn faster and are less likely to break down during training.
A layerlayer, n.: One stage in a neural network: a group of artificial neurons that all take the same input, transform it, and pass the result on to the next stage. that standardizes intermediate activationsactivation, n.: The numbers a neural network computes inside itself as it processes an input — a reading of which of its artificial neurons are firing, and how strongly, at that moment., typically subtracting a mean and dividing by a standard deviation computed over some set of values and then applying a learned scale and shift, to keep their distribution stable during training. Variants differ in the set used for the statistics: batch normalizationbatch normalization, n.: A layer that rescales the numbers flowing through a network so that, across each batch of training examples, they keep a steady average and spread. It makes deep networks train faster and more reliably. (each channel across a mini-batch), layer normalizationlayer normalization, n.: A layer that rescales the numbers inside a network, separately for each example, so they keep a steady average and spread. It, or a simpler variant, is a standard part of the models behind today's chatbots. (all features of one example), instance normalization (each channel of one example, common in style transferstyle transfer, n.: Repainting a photo in the style of an artwork — your street as Van Gogh might have painted it — by keeping the photo's content and borrowing the painting's colors, strokes and textures.), group normalization (groups of channels of one example, robust to small batches) and RMSNormRMSNorm, n.: A simpler, faster version of the layer that keeps a network's internal numbers at a steady scale. Most recent large language models use it. (scale only, no centering). Weight normalization and spectral normalization instead reparameterize or constrain the weights. Normalization permits higher learning rates and less careful initialization and, together with residual connectionsresidual connection, n.: A shortcut in a neural network that adds a block's input straight to its output, so each block only has to learn a correction. It is what makes networks hundreds of layers deep trainable., is what lets very deep networks train; its placement before or after the residual addition strongly affects stability.
With only two images per GPU, we switched from batch normalization to group normalization.
Origin
Local response normalization appeared in AlexNet (2012), but the modern family began with Sergey Ioffe and Christian Szegedy's batch normalization (2015). Weight normalization (Tim Salimans and Diederik Kingma), layer normalization (Jimmy Ba and colleagues) and instance normalization (Dmitry Ulyanov and colleagues) followed in 2016, group normalization (Yuxin Wu and Kaiming He) in 2018 and RMSNorm in 2019.
Usage note
Distinct from normalizing input data before training (feature scalingfeature scaling, n.: Putting input measurements on comparable scales, so that income in dollars and age in years, say, both span similar ranges and no feature dominates just because of its units.) and from turning scores into probabilities (softmaxsoftmax, n.: A formula that turns a list of raw scores into probabilities — all positive and adding up to one — with the biggest share going to the highest score.).
Related
- batch normalizationbatch normalization, n.: A layer that rescales the numbers flowing through a network so that, across each batch of training examples, they keep a steady average and spread. It makes deep networks train faster and more reliably.
- layer normalizationlayer normalization, n.: A layer that rescales the numbers inside a network, separately for each example, so they keep a steady average and spread. It, or a simpler variant, is a standard part of the models behind today's chatbots.
- RMSNormRMSNorm, n.: A simpler, faster version of the layer that keeps a network's internal numbers at a steady scale. Most recent large language models use it.
- residual connectionresidual connection, n.: A shortcut in a neural network that adds a block's input straight to its output, so each block only has to learn a correction. It is what makes networks hundreds of layers deep trainable.
- feature scalingfeature scaling, n.: Putting input measurements on comparable scales, so that income in dollars and age in years, say, both span similar ranges and no feature dominates just because of its units.
- training instabilitytraining instability, n.: The tendency of very large training runs to go wrong — the error suddenly jumping or blowing up — which can waste weeks of expensive computing unless engineers prevent or repair it.
Mentioned in
layerlayer, n.: One stage in a neural network: a group of artificial neurons that all take the same input, transform it, and pass the result on to the next stage., batch normalizationbatch normalization, n.: A layer that rescales the numbers flowing through a network so that, across each batch of training examples, they keep a steady average and spread. It makes deep networks train faster and more reliably., exploding gradient problemexploding gradient problem, n.: A training failure in which the learning signal grows out of control as it passes back through a network, so the weights jump wildly and training breaks down., layer normalizationlayer normalization, n.: A layer that rescales the numbers inside a network, separately for each example, so they keep a steady average and spread. It, or a simpler variant, is a standard part of the models behind today's chatbots., vanishing gradient problemvanishing gradient problem, n.: A training difficulty in deep or recurrent networks: the learning signal shrinks as it travels back through many layers or time steps, so the early parts of the network barely learn., RMSNormRMSNorm, n.: A simpler, faster version of the layer that keeps a network's internal numbers at a steady scale. Most recent large language models use it.