Skip to content
Last Dictionary
Plane
Deep LearningTraining & Post-training

backpropagation through time

nounAlso: BPTT, truncated backpropagation through time, truncated BPTT

In plain wordsThe way recurrent networks are trained: the network is unrolled into a long chain, one copy per time step, and errors are traced back through every step to adjust the shared weights.

  1. The application of backpropagationbackpropagation, n.: The method neural networks use to learn from mistakes: it works backward from the error to figure out how much each internal connection contributed, so every connection can be adjusted in the right direction. to a recurrent neural networkrecurrent neural network, n.: A neural network that reads a sequence, such as a sentence or an audio clip, one piece at a time, carrying a running summary forward so that what came earlier can shape how later pieces are understood. by unrolling it over the steps of a sequence into a deep feedforward neural networkfeedforward neural network, n.: A neural network laid out like an assembly line: each stage hands its result only to later stages, never back, so every answer depends on the current input alone and nothing is remembered between inputs. whose layers share the same weights, then propagating errors backward from the losses at each step; the gradient for each shared weight is the sum of its contributions across all steps. Memory grows linearly with sequence length, because every intermediate state must be stored, and gradients through long chains are prone to vanishingvanishing gradient problem, n.: A training difficulty in deep or recurrent networks: the learning signal shrinks as it travels back through many layers or time steps, so the early parts of the network barely learn. or explodingexploding gradient problem, n.: A training failure in which the learning signal grows out of control as it passes back through a network, so the weights jump wildly and training breaks down.. Truncated BPTT limits the backward pass to a window of k steps while still carrying the hidden state forward, trading long-range credit assignment for cost. Real-time recurrent learning (RTRL) computes the same gradients forward in time, online, but at far higher computational cost.

    We truncate backpropagation through time at 128 steps but carry the hidden state across batches.

Origin

Derived independently by several researchers in the 1980s. David Rumelhart, Geoffrey Hinton and Ronald Williams described unfolding a recurrent network in time (1986), Anthony Robinson and Frank Fallside gave a version in 1987, and Paul Werbos, who applied it in 1988, popularized the name with 'Backpropagation Through Time: What It Does and How to Do It' (Proceedings of the IEEE, 1990).

Related

See also
  • backpropagationbackpropagation, n.: The method neural networks use to learn from mistakes: it works backward from the error to figure out how much each internal connection contributed, so every connection can be adjusted in the right direction.
  • recurrent neural networkrecurrent neural network, n.: A neural network that reads a sequence, such as a sentence or an audio clip, one piece at a time, carrying a running summary forward so that what came earlier can shape how later pieces are understood.
  • vanishing gradient problemvanishing gradient problem, n.: A training difficulty in deep or recurrent networks: the learning signal shrinks as it travels back through many layers or time steps, so the early parts of the network barely learn.
  • exploding gradient problemexploding gradient problem, n.: A training failure in which the learning signal grows out of control as it passes back through a network, so the weights jump wildly and training breaks down.
  • long short-term memorylong short-term memory, n.: A type of recurrent network with a separate memory lane and learned 'gates' that decide what to remember, forget and output, letting it keep track of information over long sequences.
  • gradient checkpointinggradient checkpointing, n.: A memory-saving trick: instead of storing every intermediate result needed for learning, the computer keeps only a few and recalculates the rest when needed, trading extra computing time for memory.

Sources

  1. Werbos, P. J. (1990). Backpropagation Through Time: What It Does and How to Do It. Proceedings of the IEEE, 78(10), 1550–1560.

Mentioned in

backpropagationbackpropagation, n.: The method neural networks use to learn from mistakes: it works backward from the error to figure out how much each internal connection contributed, so every connection can be adjusted in the right direction., long short-term memorylong short-term memory, n.: A type of recurrent network with a separate memory lane and learned 'gates' that decide what to remember, forget and output, letting it keep track of information over long sequences., recurrent neural networkrecurrent neural network, n.: A neural network that reads a sequence, such as a sentence or an audio clip, one piece at a time, carrying a running summary forward so that what came earlier can shape how later pieces are understood., vanishing gradient problemvanishing gradient problem, n.: A training difficulty in deep or recurrent networks: the learning signal shrinks as it travels back through many layers or time steps, so the early parts of the network barely learn., Paul WerbosPaul Werbos, prop. n.: An American scientist whose 1974 doctoral thesis set out backpropagation, the method of passing errors backward that now trains nearly all neural networks., reservoir computingreservoir computing, n.: A shortcut way to build a network for time-varying signals: a large, fixed, randomly wired network echoes the input, and only a simple final readout is trained., Sepp HochreiterSepp Hochreiter, prop. n.: A German computer scientist who explained why early neural networks struggled to learn long sequences and, with Jürgen Schmidhuber, invented the LSTM network that solved the problem.