Skip to content
Last Dictionary
Plane
Deep LearningGenerative MediaMathematics & Statistics

energy-based model

nounAlso: EBM, energy function, energy-based learning

In plain wordsA model that scores how well things fit together by giving each possibility an 'energy': low for plausible combinations, high for implausible ones, like a ball settling into the lowest valleys.

  1. A model that scores each possible configuration of its variables with a single number, the energy E_θ(x) or E_θ(x, y): low for plausible or mutually compatible values, high for implausible ones. In the framing of Yann LeCun and colleagues' 2006 tutorial, inference is then a search for low-energy configurations, such as the label y that minimizes E(x, y). A probabilistic EBM turns energies into a Boltzmann (Gibbs) distribution whose normalizing constant, the partition function, is usually intractable, so training relies on Markov chain Monte CarloMarkov chain Monte Carlo, n.: A family of methods for drawing random examples from a complicated probability distribution by taking a long, carefully designed random walk that eventually visits each region in proportion to its probability. sampling (contrastive divergence, Langevin dynamics), score matching, noise-contrastive estimation or contrastive losses that push energy down on data and up elsewhere. Hopfield networksHopfield network, n.: A network of on–off units, all connected to one another, that stores patterns as stable states. Given a partial or noisy pattern, it settles into the closest stored one, like recalling a memory from a hint., Boltzmann machinesBoltzmann machine, n.: A network of randomly flipping on–off units, including hidden ones, that learns the statistical patterns in its training data and can then produce new examples like them. and RBMsrestricted Boltzmann machine, n.: A two-layer network, one layer for the data and one hidden, that learns the patterns in data by trying to generate it. Stacks of them helped start the deep-learning revival around 2006. are classic EBMs; modern ones use deep networks as energy functions, and any softmax classifier can be read as one. Score-basedscore-based generative model, n.: A generator that learns, for any noisy example, which direction would make it look more like real data — then starts from pure noise and follows those directions step by step. and diffusion modelsdiffusion model, n.: An AI that learns to create images, sound or video by practicing noise removal: shown pictures buried under static, it learns to clean them up, then builds new ones starting from pure static. learn the gradient of a log-density, which is the negative gradient of an energy.

    Sampling from the energy-based model took 60 Langevin steps per image.

pθ(x)=exp⁡(−Eθ(x))Z(θ),Z(θ)=∫exp⁡(−Eθ(x)) dxp_\theta(x) = \frac{\exp(-E_\theta(x))}{Z(\theta)}, \qquad Z(\theta) = \int \exp(-E_\theta(x))\,dx

Origin

Energy functions entered neural networks from statistical physics with John Hopfield's 1982 network and the Boltzmann machine of David Ackley, Geoffrey Hinton and Terrence Sejnowski (1985). The phrase 'energy-based models' appears in a 2003 paper by Yee Whye Teh, Max Welling, Simon Osindero and Hinton; Yann LeCun and colleagues' 'A Tutorial on Energy-Based Learning' (2006) extended it to non-probabilistic models.

Usage note

LeCun advocates non-generative, energy-based joint-embedding architecturesjoint-embedding predictive architecture, n.: A way of training AI on images or video in which the model predicts an abstract summary of a hidden part, rather than every pixel, so it learns what matters and ignores unpredictable detail. as an alternative to likelihood-based generative models; others regard EBMs as hard to train and sample at scale.

Related

See also
  • Boltzmann machineBoltzmann machine, n.: A network of randomly flipping on–off units, including hidden ones, that learns the statistical patterns in its training data and can then produce new examples like them.
  • restricted Boltzmann machinerestricted Boltzmann machine, n.: A two-layer network, one layer for the data and one hidden, that learns the patterns in data by trying to generate it. Stacks of them helped start the deep-learning revival around 2006.
  • Hopfield networkHopfield network, n.: A network of on–off units, all connected to one another, that stores patterns as stable states. Given a partial or noisy pattern, it settles into the closest stored one, like recalling a memory from a hint.
  • score-based generative modelscore-based generative model, n.: A generator that learns, for any noisy example, which direction would make it look more like real data — then starts from pure noise and follows those directions step by step.
  • joint-embedding predictive architecturejoint-embedding predictive architecture, n.: A way of training AI on images or video in which the model predicts an abstract summary of a hidden part, rather than every pixel, so it learns what matters and ignores unpredictable detail.
  • contrastive learningcontrastive learning, n.: A way to learn without labels by comparison: the model learns to pull together two versions of the same thing, such as two crops of one photo, and push apart unrelated things.
  • Markov chain Monte CarloMarkov chain Monte Carlo, n.: A family of methods for drawing random examples from a complicated probability distribution by taking a long, carefully designed random walk that eventually visits each region in proportion to its probability.

Sources

  1. LeCun, Y., Chopra, S., Hadsell, R., Ranzato, M. & Huang, F. J. (2006). A Tutorial on Energy-Based Learning. In Predicting Structured Data. MIT Press.

Mentioned in

Boltzmann machineBoltzmann machine, n.: A network of randomly flipping on–off units, including hidden ones, that learns the statistical patterns in its training data and can then produce new examples like them., entropyentropy, n.: A measure of unpredictability. A fair coin toss has more entropy than a rigged one; an AI that is unsure of its next word has high entropy., Hopfield networkHopfield network, n.: A network of on–off units, all connected to one another, that stores patterns as stable states. Given a partial or noisy pattern, it settles into the closest stored one, like recalling a memory from a hint., John HopfieldJohn Hopfield, prop. n.: An American physicist who showed in 1982 how a network of simple units can store and recall memories, and who shared the 2024 Nobel Prize in Physics for work that enabled machine learning., associative memoryassociative memory, n.: A memory that is searched by content rather than by location: given part of a stored pattern, or a noisy copy of it, it recalls the whole, much as a few notes can bring back an entire song., Markov chain Monte CarloMarkov chain Monte Carlo, n.: A family of methods for drawing random examples from a complicated probability distribution by taking a long, carefully designed random walk that eventually visits each region in proportion to its probability., Markov random fieldMarkov random field, n.: A model of many linked uncertain quantities, such as neighboring pixels in a photo, in which each one depends directly only on its neighbors, with no notion of which one causes which., restricted Boltzmann machinerestricted Boltzmann machine, n.: A two-layer network, one layer for the data and one hidden, that learns the patterns in data by trying to generate it. Stacks of them helped start the deep-learning revival around 2006., score-based generative modelscore-based generative model, n.: A generator that learns, for any noisy example, which direction would make it look more like real data — then starts from pure noise and follows those directions step by step.