Skip to content
Last Dictionary
Plane
Machine LearningHistory & LandmarksDeep Learning

delta rule

nounAlso: Widrow–Hoff rule, least mean squares rule, LMS algorithm, generalized delta rule

In plain wordsA simple learning rule for one artificial neuron: adjust each connection in proportion to the error, the gap between the wanted and the actual output, times that connection's input.

  1. A learning rule that updates each weight of a linear unit in proportion to the error δ, the difference between the target and the unit's output, multiplied by the corresponding input and a learning ratelearning rate, n.: How big a step a model takes each time it adjusts its settings during training. Too large and learning goes haywire; too small and it crawls.. It is stochastic gradient descentstochastic gradient descent, n.: A faster version of gradient descent that estimates which way is downhill from a small random sample of the training data at each step, instead of checking the whole dataset. on the squared error of a single linear unit, so with a small enough learning rate it converges toward the least-squares solution. Unlike the perceptronperceptron, n.: One of the first learning machines: a single artificial neuron that sorts inputs into two groups by drawing a straight dividing line, nudging its weights whenever it guesses wrong. rule, which uses the thresholded output and changes weights only after a misclassification, the delta rule uses the continuous output and keeps adjusting until the error is minimized. It trained ADALINEADALINE, prop. n.: An early learning machine from 1960 that adjusted its settings to shrink its errors step by step, the same basic idea used to train today's neural networks. and, as the least-mean-squares (LMS) algorithm, became a workhorse of adaptive signal processing in echo cancellers and channel equalizers. Its extension to multilayer networks with differentiable activations, which passes δ terms backward through the layers, was called the generalized delta rule and is backpropagationbackpropagation, n.: The method neural networks use to learn from mistakes: it works backward from the error to figure out how much each internal connection contributed, so every connection can be adjusted in the right direction..

    Under the delta rule, the weight on an input that was strongly active during a large error moves the most.

Δwi=η (t−y) xi\Delta w_i = \eta\,(t - y)\,x_i

Origin

Introduced by Bernard Widrow and his student Marcian (Ted) Hoff at Stanford in 1960 as the least-mean-squares rule for training ADALINE. The name 'delta rule' became standard through the parallel distributed processing work of the 1980s, in which David Rumelhart, Geoffrey Hinton and Ronald Williams (1986) called backpropagation the 'generalized delta rule'; psychology's Rescorla–Wagner model of conditioning (1972) uses the same error-driven update.

Usage note

Often conflated with the perceptron learning rule, which has the same form but uses the thresholded output and therefore behaves differently when the data are not linearly separable.

Related

See also
  • ADALINEADALINE, prop. n.: An early learning machine from 1960 that adjusted its settings to shrink its errors step by step, the same basic idea used to train today's neural networks.
  • perceptronperceptron, n.: One of the first learning machines: a single artificial neuron that sorts inputs into two groups by drawing a straight dividing line, nudging its weights whenever it guesses wrong.
  • stochastic gradient descentstochastic gradient descent, n.: A faster version of gradient descent that estimates which way is downhill from a small random sample of the training data at each step, instead of checking the whole dataset.
  • backpropagationbackpropagation, n.: The method neural networks use to learn from mistakes: it works backward from the error to figure out how much each internal connection contributed, so every connection can be adjusted in the right direction.
  • learning ratelearning rate, n.: How big a step a model takes each time it adjusts its settings during training. Too large and learning goes haywire; too small and it crawls.
  • Hebbian learningHebbian learning, n.: The idea that a connection between two brain cells grows stronger when they are active at the same time — the principle behind the slogan 'cells that fire together wire together'.
Contrast with
  • perceptronperceptron, n.: One of the first learning machines: a single artificial neuron that sorts inputs into two groups by drawing a straight dividing line, nudging its weights whenever it guesses wrong.
  • Hebbian learningHebbian learning, n.: The idea that a connection between two brain cells grows stronger when they are active at the same time — the principle behind the slogan 'cells that fire together wire together'.

Sources

  1. Widrow, B. & Hoff, M. E. (1960). Adaptive Switching Circuits. IRE WESCON Convention Record, Part 4, 96–104.
  2. Rumelhart, D. E., Hinton, G. E. & Williams, R. J. (1986). Learning Internal Representations by Error Propagation. In Parallel Distributed Processing, Vol. 1. MIT Press.