Parallel Distributed Processing
In plain wordsA two-volume 1986 book by a group of psychologists and computer scientists that revived interest in brain-like neural networks and made the training method behind modern AI widely known.
A two-volume work, Explorations in the Microstructure of Cognition (MIT Press, 1986), edited by David RumelhartDavid Rumelhart, prop. n.: An American psychologist who, with Geoffrey Hinton and Ronald Williams, showed in 1986 how to train multilayer neural networks, launching the modern revival of the field., James McClelland and the PDP Research Group, and by extension the modeling framework it set out. The books argued that cognition arises from many simple processing units interacting in parallel, with knowledge distributed across connection weights, and presented models of perception, memory, language and learning. Chapter 8 of the first volume, by Rumelhart, Geoffrey HintonGeoffrey Hinton, prop. n.: A British-Canadian scientist whose decades of persistent work on brain-inspired neural networks laid the foundations of modern AI; often called a 'godfather of AI', he now warns publicly about its dangers. and Ronald Williams, gave the standard account of training multilayer networks by backpropagationbackpropagation, n.: The method neural networks use to learn from mistakes: it works backward from the error to figure out how much each internal connection contributed, so every connection can be adjusted in the right direction. of errors, and the second volume included a much-debated model of how children learn the English past tense. The volumes became founding texts of the 1980s revival of connectionismconnectionism, n.: The view that minds and intelligent machines are best built from huge networks of simple, neuron-like units whose connection strengths are learned, rather than from explicit rules and symbols.; a third volume of exercises and software followed in 1988.
Many deep-learning pioneers of the 2010s first met backpropagation in the Parallel Distributed Processing volumes.
Origin
The PDP Research Group formed around 1981 at the University of California, San Diego; contributors included Rumelhart, McClelland, Hinton, Paul Smolensky, Michael Jordan and Terrence Sejnowski. The backpropagation chapter appeared alongside the 1986 Nature paper by Rumelhart, Hinton and Williams, 'Learning Representations by Back-Propagating Errors'.
Usage note
Not to be confused with DEC's PDP (Programmed Data Processor) minicomputers. Backpropagation itself had earlier precursors, notably Seppo Linnainmaa (1970) and Paul Werbos (1974); see backpropagationbackpropagation, n.: The method neural networks use to learn from mistakes: it works backward from the error to figure out how much each internal connection contributed, so every connection can be adjusted in the right direction..
Related
- connectionismconnectionism, n.: The view that minds and intelligent machines are best built from huge networks of simple, neuron-like units whose connection strengths are learned, rather than from explicit rules and symbols.
- backpropagationbackpropagation, n.: The method neural networks use to learn from mistakes: it works backward from the error to figure out how much each internal connection contributed, so every connection can be adjusted in the right direction.
- David RumelhartDavid Rumelhart, prop. n.: An American psychologist who, with Geoffrey Hinton and Ronald Williams, showed in 1986 how to train multilayer neural networks, launching the modern revival of the field.
- Geoffrey HintonGeoffrey Hinton, prop. n.: A British-Canadian scientist whose decades of persistent work on brain-inspired neural networks laid the foundations of modern AI; often called a 'godfather of AI', he now warns publicly about its dangers.
- multilayer perceptronmultilayer perceptron, n.: A neural network made of several layers of artificial neurons, each connected to every neuron in the next layer. It is the simplest kind of deep network and a component of almost every modern model.
- Perceptrons (book)Perceptrons (book), prop. n.: A 1969 book by Marvin Minsky and Seymour Papert proving that simple one-layer neural networks cannot learn certain patterns. It is widely blamed for a decade-long slump in neural-network research.
Sources
- Rumelhart, D. E., McClelland, J. L. & the PDP Research Group (1986). Parallel Distributed Processing: Explorations in the Microstructure of Cognition. 2 vols. Cambridge, MA: MIT Press.
- Rumelhart, D. E., Hinton, G. E. & Williams, R. J. (1986). Learning Representations by Back-Propagating Errors. Nature 323, 533–536.
Mentioned in
connectionismconnectionism, n.: The view that minds and intelligent machines are best built from huge networks of simple, neuron-like units whose connection strengths are learned, rather than from explicit rules and symbols., David RumelhartDavid Rumelhart, prop. n.: An American psychologist who, with Geoffrey Hinton and Ronald Williams, showed in 1986 how to train multilayer neural networks, launching the modern revival of the field., Perceptrons (book)Perceptrons (book), prop. n.: A 1969 book by Marvin Minsky and Seymour Papert proving that simple one-layer neural networks cannot learn certain patterns. It is widely blamed for a decade-long slump in neural-network research.