Skip to content
Last Dictionary
Plane
Deep LearningTraining & Post-training

batch normalization

nounAlso: BatchNorm, batch norm, BN, internal covariate shift

In plain wordsA layer that rescales the numbers flowing through a network so that, across each batch of training examples, they keep a steady average and spread. It makes deep networks train faster and more reliably.

  1. A normalization layernormalization layer, n.: A layer that keeps the numbers passing through a neural network within a steady range by shifting and resizing them, so that deep networks learn faster and are less likely to break down during training. that, for each feature or channel, subtracts the mean and divides by the standard deviation computed over the current mini-batchmini-batch, n.: A small random handful of training examples that a model learns from in a single step. Training works through many such handfuls instead of studying the whole dataset at once. (and, in convolutional networks, over spatial positions), then applies a learned scale γ and shift β. Training uses batch statistics; inference uses running averages accumulated during training, so behavior differs between the two modes. It permits much higher learning rateslearning rate, n.: How big a step a model takes each time it adjusts its settings during training. Too large and learning goes haywire; too small and it crawls., reduces sensitivity to initializationweight initialization, n.: Choosing the random starting values of a neural network's internal numbers before training. Good starting values keep signals from fading away or blowing up as they pass through many layers., adds mild regularizing noise, and became standard in CNNsconvolutional neural network, n.: A kind of neural network built for images and other grid-like data. It slides small pattern detectors across the input, finding edges first, then shapes, then whole objects. such as ResNetResNet, prop. n.: A family of image-recognition networks from 2015 whose shortcut connections allowed far deeper networks than before, over 100 layers, and became a standard building block across AI.. Its dependence on batch statistics is also its weakness: it degrades with small or unrepresentative batches, couples examples within a batch, complicates distributed training and suits sequence models poorly, so transformerstransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it. use layer normalizationlayer normalization, n.: A layer that rescales the numbers inside a network, separately for each example, so they keep a steady average and spread. It, or a simpler variant, is a standard part of the models behind today's chatbots. or RMSNormRMSNorm, n.: A simpler, faster version of the layer that keeps a network's internal numbers at a steady scale. Most recent large language models use it. instead. Its inventors credited it with reducing 'internal covariate shift', the drift in each layer's input distribution during training; later work disputes this and points instead to a smoother loss landscapeloss landscape, n.: A way of picturing training as a journey across hilly terrain, where position stands for the model's settings and height for its error. Training searches for low valleys., so the mechanism remains debated.

    Remember to call model.eval() so batch normalization uses its running statistics at test time.

x^=x−μBσB2+ϵ,y=γx^+β\hat{x} = \frac{x - \mu_{\mathcal{B}}}{\sqrt{\sigma_{\mathcal{B}}^2 + \epsilon}}, \qquad y = \gamma \hat{x} + \beta

Origin

Introduced by Sergey Ioffe and Christian Szegedy of Google in 'Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift' (arXiv, February 2015; ICML 2015), which reported matching an ImageNet model's accuracy with 14 times fewer training steps. Shibani Santurkar and colleagues (2018) argued that its benefit comes mainly from smoothing the optimization landscape.

Related

See also
  • layer normalizationlayer normalization, n.: A layer that rescales the numbers inside a network, separately for each example, so they keep a steady average and spread. It, or a simpler variant, is a standard part of the models behind today's chatbots.
  • RMSNormRMSNorm, n.: A simpler, faster version of the layer that keeps a network's internal numbers at a steady scale. Most recent large language models use it.
  • normalization layernormalization layer, n.: A layer that keeps the numbers passing through a neural network within a steady range by shifting and resizing them, so that deep networks learn faster and are less likely to break down during training.
  • mini-batchmini-batch, n.: A small random handful of training examples that a model learns from in a single step. Training works through many such handfuls instead of studying the whole dataset at once.
  • ResNetResNet, prop. n.: A family of image-recognition networks from 2015 whose shortcut connections allowed far deeper networks than before, over 100 layers, and became a standard building block across AI.
  • loss landscapeloss landscape, n.: A way of picturing training as a journey across hilly terrain, where position stands for the model's settings and height for its error. Training searches for low valleys.

Sources

  1. Ioffe, S. & Szegedy, C. (2015). Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. arXiv:1502.03167; ICML 2015. arxiv.org/abs/1502.03167
  2. Santurkar, S., Tsipras, D., Ilyas, A. & Madry, A. (2018). How Does Batch Normalization Help Optimization? arXiv:1805.11604; NeurIPS 2018. arxiv.org/abs/1805.11604

Mentioned in

dropoutdropout, n.: A training trick that randomly switches off a fraction of a network's neurons at each step, so the network cannot lean too heavily on any one of them and learns sturdier patterns., feature scalingfeature scaling, n.: Putting input measurements on comparable scales, so that income in dollars and age in years, say, both span similar ranges and no feature dominates just because of its units., gradient accumulationgradient accumulation, n.: A way to train with large batches on limited hardware: the computer processes several small batches one after another, adds up what each says, and only then makes a single update., ICMLICML, prop. n.: A leading machine learning conference, first held as a workshop in 1980, where researchers gather each summer to present new learning methods and theory., layer normalizationlayer normalization, n.: A layer that rescales the numbers inside a network, separately for each example, so they keep a steady average and spread. It, or a simpler variant, is a standard part of the models behind today's chatbots., mini-batchmini-batch, n.: A small random handful of training examples that a model learns from in a single step. Training works through many such handfuls instead of studying the whole dataset at once., normalization layernormalization layer, n.: A layer that keeps the numbers passing through a neural network within a steady range by shifting and resizing them, so that deep networks learn faster and are less likely to break down during training., ResNetResNet, prop. n.: A family of image-recognition networks from 2015 whose shortcut connections allowed far deeper networks than before, over 100 layers, and became a standard building block across AI., variancevariance, n.: A number that grows the more widely something's possible values scatter away from its average, and is zero if it never varies. Its square root, the standard deviation, uses the same units as the values themselves., weight decayweight decay, n.: A training rule that shrinks every weight slightly at each step, like a small tax on large numbers. It keeps models simpler and usually helps them work better on new data., ConvNeXtConvNeXt, prop. n.: A 2022 image-recognition network showing that an older-style convolutional design, updated with lessons from transformers, can match vision transformers., InceptionInception, prop. n.: A family of Google image-recognition networks, the first called GoogLeNet, whose building blocks examine each patch of an image at several scales at once while keeping computation low., RMSNormRMSNorm, n.: A simpler, faster version of the layer that keeps a network's internal numbers at a steady scale. Most recent large language models use it.