Skip to content
Last Dictionary
Plane
Generative MediaDeep LearningMathematics & Statistics

normalizing flow

nounAlso: flow-based generative model, flow model, invertible neural network, continuous normalizing flow, RealNVP, Glow

In plain wordsA generator built from a chain of reversible steps that reshape simple random noise into realistic data; because every step can be undone, it can also calculate exactly how likely any example is.

  1. A generative modelgenerative model, n.: A model that learns what a kind of data looks like — faces, sentences, songs — well enough to produce new examples that resemble the originals without copying them. built from a chain of invertible, differentiable functions that reshape a simple base distribution, typically a standard Gaussian, into the data distribution. Sampling pushes base noise forward through the chain; density evaluation runs data backward and applies the change-of-variables formula, adding the log-absolute-determinant of the JacobianJacobian, n.: A table listing how each output of a process responds to a small change in each input. It describes how a transformation locally stretches, squeezes and turns space. of each step. Because every step can be undone and its change of volume computed, flows give exact likelihoods and exact latent codes, which VAEsvariational autoencoder, n.: A neural network that learns to squeeze data into a compact, slightly fuzzy code and rebuild it, arranged so that decoding random codes produces new, plausible examples. (which only bound the likelihood) and GANsgenerative adversarial network, n.: Two neural networks trained as rivals: one forges fake images, the other tries to spot the fakes. As both improve, the forger's output becomes realistic enough to fool people. (which have none) do not. The price is architectural: every layer must be invertible with a cheap Jacobian determinant, as in coupling layers (NICE, RealNVP, Glow) and autoregressive flows (MAF, IAF). Continuous normalizing flows instead define the transformation as the solution of a learned ordinary differential equation (FFJORD, a neural ordinary differential equationneural ordinary differential equation, n.: A neural network that, instead of passing data through a fixed stack of layers, describes how the data changes continuously over 'time' and uses an equation solver to follow that change.); training them without simulation by flow matchingflow matching, n.: A way to train generators that learns, for every point between random noise and real data, which direction to move, so that noise can be steered smoothly into new examples. links flows to modern diffusiondiffusion model, n.: An AI that learns to create images, sound or video by practicing noise removal: shown pictures buried under static, it learns to clean them up, then builds new ones starting from pure static. practice.

log⁡pX(x)=log⁡pZ(f(x))+log⁡∣det⁡∂f(x)∂x∣\log p_X(x) = \log p_Z\big(f(x)\big) + \log \left| \det \frac{\partial f(x)}{\partial x} \right|

Origin

Esteban Tabak and Eric Vanden-Eijnden (2010) estimated densities by flowing data step by step toward a normal distribution; Tabak and Cristina Turner (2013) introduced the term 'normalizing flow' for such a composition of simple maps. Oren Rippel and Ryan Adams (2013) were among the first to build the maps from deep networks, and Danilo Rezende and Shakir Mohamed brought the term into deep learning with 'Variational Inference with Normalizing Flows' (posted May 2015); NICE (2014), RealNVP (2016) and Glow (2018) made flows practical image generators.

Usage note

'Normalizing' refers to transforming data toward a normal (Gaussian) distribution, not to normalizing probabilities so they sum to one.

Related

See also
  • generative modelgenerative model, n.: A model that learns what a kind of data looks like — faces, sentences, songs — well enough to produce new examples that resemble the originals without copying them.
  • flow matchingflow matching, n.: A way to train generators that learns, for every point between random noise and real data, which direction to move, so that noise can be steered smoothly into new examples.
  • variational autoencodervariational autoencoder, n.: A neural network that learns to squeeze data into a compact, slightly fuzzy code and rebuild it, arranged so that decoding random codes produces new, plausible examples.
  • JacobianJacobian, n.: A table listing how each output of a process responds to a small change in each input. It describes how a transformation locally stretches, squeezes and turns space.
  • neural ordinary differential equationneural ordinary differential equation, n.: A neural network that, instead of passing data through a fixed stack of layers, describes how the data changes continuously over 'time' and uses an equation solver to follow that change.
  • density estimationdensity estimation, n.: Working out from a sample of data how likely different values are, by drawing a smooth curve or surface showing where data points bunch up and where they are rare.
  • diffusion modeldiffusion model, n.: An AI that learns to create images, sound or video by practicing noise removal: shown pictures buried under static, it learns to clean them up, then builds new ones starting from pure static.

Sources

  1. Rezende, D. J. & Mohamed, S. (2015). Variational Inference with Normalizing Flows. arXiv:1505.05770 (ICML 2015). arxiv.org/abs/1505.05770
  2. Papamakarios, G., Nalisnick, E., Rezende, D. J., Mohamed, S. & Lakshminarayanan, B. (2021). Normalizing Flows for Probabilistic Modeling and Inference. Journal of Machine Learning Research 22. arXiv:1912.02762. arxiv.org/abs/1912.02762

Mentioned in

density estimationdensity estimation, n.: Working out from a sample of data how likely different values are, by drawing a smooth curve or surface showing where data points bunch up and where they are rare., flow matchingflow matching, n.: A way to train generators that learns, for every point between random noise and real data, which direction to move, so that noise can be steered smoothly into new examples., generative modelgenerative model, n.: A model that learns what a kind of data looks like — faces, sentences, songs — well enough to produce new examples that resemble the originals without copying them., Diederik KingmaDiederik Kingma, prop. n.: A Dutch computer scientist who co-invented the variational autoencoder and the Adam optimizer, two of the most widely used tools for training neural networks., JacobianJacobian, n.: A table listing how each output of a process responds to a small change in each input. It describes how a transformation locally stretches, squeezes and turns space., neural ordinary differential equationneural ordinary differential equation, n.: A neural network that, instead of passing data through a fixed stack of layers, describes how the data changes continuously over 'time' and uses an equation solver to follow that change., probability density functionprobability density function, n.: A curve showing where a measurement is likely to fall. The area under the curve between two values gives the chance of landing between them.