deep belief network
In plain wordsA many-layered neural network trained one layer at a time, introduced in 2006. It showed that deep networks could be trained after all and helped launch the deep learning boom, before simpler methods replaced it.
A generative model assembled by stacking restricted Boltzmann machinesBoltzmann machine, n.: A network of randomly flipping on–off units, including hidden ones, that learns the statistical patterns in its training data and can then produce new examples like them.: once training is done, only the topmost pair of hidden layers keeps its two-way connections, while every connection beneath it becomes a one-way, top-down path that turns hidden causes into data. Training proceeds greedily, one layer at a time: the first restricted Boltzmann machine models the data, the activities of its hidden units serve as data for the second, and so on, each added layer improving a bound on the likelihood of the data. The stack can then be fine-tuned generatively or used to initialize a discriminative network trained with backpropagationbackpropagation, n.: The method neural networks use to learn from mistakes: it works backward from the error to figure out how much each internal connection contributed, so every connection can be adjusted in the right direction.. This unsupervised pretrainingpretraining, n.: The first and usually the most expensive stage of building a modern AI model, in which it learns general patterns from enormous amounts of raw text, images or code before being specialized for particular uses. made networks with many layers trainable at a time when they were widely thought impractical, and set records on MNISTMNIST, prop. n.: A famous collection of 70,000 small images of handwritten digits used to test image-recognition programs. Getting a model to read MNIST digits is the 'hello world' of machine learning. and in speech recognition. By about 2012 it had largely been abandoned, as better initialization, ReLUReLU, abbr.: The most common rule a neuron uses to produce its output: keep the number if it is positive, replace it with zero if it is negative. Simple, fast and surprisingly effective. activations, more data and GPUs allowed deep networks to be trained directly.
The 2006 deep belief network papers are often taken as the opening of the deep-learning era.
Origin
Introduced by Geoffrey HintonGeoffrey Hinton, prop. n.: A British-Canadian scientist whose decades of persistent work on brain-inspired neural networks laid the foundations of modern AI; often called a 'godfather of AI', he now warns publicly about its dangers., Simon Osindero and Yee-Whye Teh in 'A Fast Learning Algorithm for Deep Belief Nets' (Neural Computation, 2006), with a companion paper by Hinton and Ruslan Salakhutdinov in Science the same year. 'Belief net' echoes the belief networks of Judea PearlJudea Pearl, prop. n.: An Israeli-American computer scientist who gave AI the mathematics for reasoning under uncertainty and, later, for telling cause and effect apart from mere correlation. and Radford Neal's sigmoid belief nets.
Related
- Boltzmann machineBoltzmann machine, n.: A network of randomly flipping on–off units, including hidden ones, that learns the statistical patterns in its training data and can then produce new examples like them.
- deep learning revolutiondeep learning revolution, n.: The rapid takeover of AI by many-layered neural networks from about 2012, when they suddenly began beating older methods at recognizing images and speech, leading to today's AI systems.
- pretrainingpretraining, n.: The first and usually the most expensive stage of building a modern AI model, in which it learns general patterns from enormous amounts of raw text, images or code before being specialized for particular uses.
- Geoffrey HintonGeoffrey Hinton, prop. n.: A British-Canadian scientist whose decades of persistent work on brain-inspired neural networks laid the foundations of modern AI; often called a 'godfather of AI', he now warns publicly about its dangers.
- deep learningdeep learning, n.: Machine learning done with neural networks built from many stacked layers: raw data such as pixels, sound or text goes in at one end, and each layer turns what it receives into something more abstract.
Sources
- Hinton, G. E., Osindero, S. & Teh, Y.-W. (2006). A Fast Learning Algorithm for Deep Belief Nets. Neural Computation 18(7), 1527–1554.
- Hinton, G. E. & Salakhutdinov, R. R. (2006). Reducing the Dimensionality of Data with Neural Networks. Science 313(5786), 504–507.
Mentioned in
Geoffrey HintonGeoffrey Hinton, prop. n.: A British-Canadian scientist whose decades of persistent work on brain-inspired neural networks laid the foundations of modern AI; often called a 'godfather of AI', he now warns publicly about its dangers., Boltzmann machineBoltzmann machine, n.: A network of randomly flipping on–off units, including hidden ones, that learns the statistical patterns in its training data and can then produce new examples like them., deep learning revolutiondeep learning revolution, n.: The rapid takeover of AI by many-layered neural networks from about 2012, when they suddenly began beating older methods at recognizing images and speech, leading to today's AI systems., godfathers of AIgodfathers of AI, phr.: A media nickname for three scientists — Geoffrey Hinton, Yoshua Bengio and Yann LeCun — whose decades of work on neural networks underpins modern AI. They shared computing's top prize in 2019., restricted Boltzmann machinerestricted Boltzmann machine, n.: A two-layer network, one layer for the data and one hidden, that learns the patterns in data by trying to generate it. Stacks of them helped start the deep-learning revival around 2006.