restricted Boltzmann machine
In plain wordsA two-layer network, one layer for the data and one hidden, that learns the patterns in data by trying to generate it. Stacks of them helped start the deep-learning revival around 2006.
A two-layer energy-based modelenergy-based model, n.: A model that scores how well things fit together by giving each possibility an 'energy': low for plausible combinations, high for implausible ones, like a ball settling into the lowest valleys. over binary or real-valued units, with a visible layer for the data and a hidden layer for features, in which every connection runs between the layers and none within them; it is the Boltzmann machineBoltzmann machine, n.: A network of randomly flipping on–off units, including hidden ones, that learns the statistical patterns in its training data and can then produce new examples like them. restricted to a bipartite graph. The restriction makes hidden units conditionally independent given the visible ones, and vice versa, so block Gibbs sampling can alternate efficiently between layers. RBMs are trained approximately by contrastive divergence, which runs a short Gibbs chain from each data point instead of sampling to equilibrium. Stacked and trained greedily layer by layer, they formed deep belief networksdeep belief network, n.: A many-layered neural network trained one layer at a time, introduced in 2006. It showed that deep networks could be trained after all and helped launch the deep learning boom, before simpler methods replaced it., used to pretrain deep networks in 2006–2011, and they were used for collaborative filtering in the Netflix PrizeNetflix Prize, prop. n.: A 2006–2009 contest in which Netflix offered a million dollars to anyone who could predict customers' movie ratings 10 percent better than its own system. It popularized machine-learning competitions and model blending.. RBM pretraining was abandoned once ReLUsReLU, abbr.: The most common rule a neuron uses to produce its output: keep the number if it is positive, replace it with zero if it is negative. Simple, fast and surprisingly effective., better initialization and large labeled datasets allowed deep networks to be trained directly.
Each layer was pretrained as an RBM for 50 epochs before fine-tuning the whole stack.
Origin
Introduced by Paul Smolensky in 1986 as the 'harmonium', in the Parallel Distributed Processing volumes. Geoffrey Hinton's contrastive divergence (2002) made it trainable, and Hinton, Simon Osindero and Yee-Whye Teh's 2006 deep-belief-network paper made stacked RBMs a centerpiece of the deep-learning revival; Ruslan Salakhutdinov, Andriy Mnih and Hinton applied them to the Netflix Prize data in 2007.
Related
- Boltzmann machineBoltzmann machine, n.: A network of randomly flipping on–off units, including hidden ones, that learns the statistical patterns in its training data and can then produce new examples like them.
- deep belief networkdeep belief network, n.: A many-layered neural network trained one layer at a time, introduced in 2006. It showed that deep networks could be trained after all and helped launch the deep learning boom, before simpler methods replaced it.
- energy-based modelenergy-based model, n.: A model that scores how well things fit together by giving each possibility an 'energy': low for plausible combinations, high for implausible ones, like a ball settling into the lowest valleys.
- Netflix PrizeNetflix Prize, prop. n.: A 2006–2009 contest in which Netflix offered a million dollars to anyone who could predict customers' movie ratings 10 percent better than its own system. It popularized machine-learning competitions and model blending.
- autoencoderautoencoder, n.: A neural network trained to squeeze its input into a compact code and then rebuild the original from that code. Learning to compress this way forces it to capture what matters in the data.
- Hopfield networkHopfield network, n.: A network of on–off units, all connected to one another, that stores patterns as stable states. Given a partial or noisy pattern, it settles into the closest stored one, like recalling a memory from a hint.
Sources
- Hinton, G. E. (2002). Training Products of Experts by Minimizing Contrastive Divergence. Neural Computation, 14(8), 1771–1800.
- Hinton, G. E., Osindero, S. & Teh, Y.-W. (2006). A Fast Learning Algorithm for Deep Belief Nets. Neural Computation, 18(7), 1527–1554.
Mentioned in
energy-based modelenergy-based model, n.: A model that scores how well things fit together by giving each possibility an 'energy': low for plausible combinations, high for implausible ones, like a ball settling into the lowest valleys.