Skip to content
Last Dictionary
Plane
History & LandmarksDeep LearningMathematics & Statistics

Boltzmann machine

nounAlso: Boltzmann machines, stochastic Hopfield network

In plain wordsA network of randomly flipping on–off units, including hidden ones, that learns the statistical patterns in its training data and can then produce new examples like them.

  1. A network of stochastic binary units with symmetric connections that defines a probability distribution over its states through an energy function, each state's probability following the Boltzmann distribution of statistical physics. Unlike a Hopfield networkHopfield network, n.: A network of on–off units, all connected to one another, that stores patterns as stable states. Given a partial or noisy pattern, it settles into the closest stored one, like recalling a memory from a hint., it has hidden units and learns: each weight is adjusted according to the difference between how often its two units are on together when the visible units are clamped to training data and when the network runs freely, which requires slow sampling such as simulated annealingsimulated annealing, n.: An optimization method inspired by slowly cooling metal: early on it sometimes accepts worse solutions, to escape dead ends, and becomes pickier as a 'temperature' setting gradually drops. or Gibbs sampling. General Boltzmann machines proved impractical to train, but the restricted Boltzmann machine (RBM), with no connections within a layer, became trainable with Geoffrey HintonGeoffrey Hinton, prop. n.: A British-Canadian scientist whose decades of persistent work on brain-inspired neural networks laid the foundations of modern AI; often called a 'godfather of AI', he now warns publicly about its dangers.'s contrastive divergence method (2002) and was the building block of the deep belief networkdeep belief network, n.: A many-layered neural network trained one layer at a time, introduced in 2006. It showed that deep networks could be trained after all and helped launch the deep learning boom, before simpler methods replaced it..

    Leading Netflix Prize teams blended restricted Boltzmann machines with matrix factorization models.

P(s)=e−E(s)/TZ,E(s)=−∑i<jwijsisj−∑ibisiP(\mathbf{s}) = \frac{e^{-E(\mathbf{s})/T}}{Z}, \qquad E(\mathbf{s}) = -\sum_{i<j} w_{ij} s_i s_j - \sum_i b_i s_i

Origin

Introduced by Geoffrey Hinton and Terrence Sejnowski in 1983 papers, one with Scott Fahlman, and set out fully in David Ackley, Hinton and Sejnowski's 'A Learning Algorithm for Boltzmann Machines' (Cognitive Science, 1985). Named after the physicist Ludwig Boltzmann. The restricted form had been described by Paul Smolensky in 1986 as the 'harmonium'.

Usage note

The 2024 Nobel Prize in Physics cited the Boltzmann machine in Hinton's share of the award; see Nobel Prizes for AINobel Prizes for AI, phr.: The two 2024 Nobel Prizes, in physics and chemistry, that honored artificial intelligence research for the first time, recognizing neural-network pioneers and the protein-structure system AlphaFold..

Related

See also
  • Hopfield networkHopfield network, n.: A network of on–off units, all connected to one another, that stores patterns as stable states. Given a partial or noisy pattern, it settles into the closest stored one, like recalling a memory from a hint.
  • deep belief networkdeep belief network, n.: A many-layered neural network trained one layer at a time, introduced in 2006. It showed that deep networks could be trained after all and helped launch the deep learning boom, before simpler methods replaced it.
  • energy-based modelenergy-based model, n.: A model that scores how well things fit together by giving each possibility an 'energy': low for plausible combinations, high for implausible ones, like a ball settling into the lowest valleys.
  • simulated annealingsimulated annealing, n.: An optimization method inspired by slowly cooling metal: early on it sometimes accepts worse solutions, to escape dead ends, and becomes pickier as a 'temperature' setting gradually drops.
  • Geoffrey HintonGeoffrey Hinton, prop. n.: A British-Canadian scientist whose decades of persistent work on brain-inspired neural networks laid the foundations of modern AI; often called a 'godfather of AI', he now warns publicly about its dangers.
  • Nobel Prizes for AINobel Prizes for AI, phr.: The two 2024 Nobel Prizes, in physics and chemistry, that honored artificial intelligence research for the first time, recognizing neural-network pioneers and the protein-structure system AlphaFold.

Sources

  1. Ackley, D. H., Hinton, G. E. & Sejnowski, T. J. (1985). A Learning Algorithm for Boltzmann Machines. Cognitive Science 9(1), 147–169.

Mentioned in

Geoffrey HintonGeoffrey Hinton, prop. n.: A British-Canadian scientist whose decades of persistent work on brain-inspired neural networks laid the foundations of modern AI; often called a 'godfather of AI', he now warns publicly about its dangers., temperaturetemperature, n.: A setting that controls how adventurous an AI model's word choices are. Low temperature makes it pick the safest, most likely words; high temperature makes it more varied and surprising, but more error-prone., connectionismconnectionism, n.: The view that minds and intelligent machines are best built from huge networks of simple, neuron-like units whose connection strengths are learned, rather than from explicit rules and symbols., Hebbian learningHebbian learning, n.: The idea that a connection between two brain cells grows stronger when they are active at the same time — the principle behind the slogan 'cells that fire together wire together'., hidden layerhidden layer, n.: Any layer of a neural network between the input and the output. Its values are never shown to the user; they are the network's internal working-out., Hopfield networkHopfield network, n.: A network of on–off units, all connected to one another, that stores patterns as stable states. Given a partial or noisy pattern, it settles into the closest stored one, like recalling a memory from a hint., John HopfieldJohn Hopfield, prop. n.: An American physicist who showed in 1982 how a network of simple units can store and recall memories, and who shared the 2024 Nobel Prize in Physics for work that enabled machine learning., Nobel Prizes for AINobel Prizes for AI, phr.: The two 2024 Nobel Prizes, in physics and chemistry, that honored artificial intelligence research for the first time, recognizing neural-network pioneers and the protein-structure system AlphaFold., associative memoryassociative memory, n.: A memory that is searched by content rather than by location: given part of a stored pattern, or a noisy copy of it, it recalls the whole, much as a few notes can bring back an entire song., deep belief networkdeep belief network, n.: A many-layered neural network trained one layer at a time, introduced in 2006. It showed that deep networks could be trained after all and helped launch the deep learning boom, before simpler methods replaced it., energy-based modelenergy-based model, n.: A model that scores how well things fit together by giving each possibility an 'energy': low for plausible combinations, high for implausible ones, like a ball settling into the lowest valleys., Markov chain Monte CarloMarkov chain Monte Carlo, n.: A family of methods for drawing random examples from a complicated probability distribution by taking a long, carefully designed random walk that eventually visits each region in proportion to its probability., restricted Boltzmann machinerestricted Boltzmann machine, n.: A two-layer network, one layer for the data and one hidden, that learns the patterns in data by trying to generate it. Stacks of them helped start the deep-learning revival around 2006., Terry SejnowskiTerry Sejnowski, prop. n.: An American neuroscientist who co-invented the Boltzmann machine with Geoffrey Hinton, built an early network that learned to read aloud, and helped found the NeurIPS conference community.