momentum

nounAlso: heavy-ball method, Polyak momentum, Nesterov momentum, Nesterov accelerated gradient, SGD with momentum

In plain wordsA training trick that lets each update keep part of the previous update's direction, like a ball rolling downhill that builds up speed and coasts over small bumps.

A modification of gradient descentgradient descent, n.: A method for training models by repeatedly nudging every setting a small step in whichever direction most reduces the error, like a hiker who cannot see the valley but keeps stepping wherever the ground tilts down. that accumulates an exponentially decaying sum of past gradients, the velocity, and steps along it instead of along the current gradient alone; the momentum coefficient β is typically 0.9. Directions in which gradients consistently agree build up speed while oscillating components cancel, which accelerates progress along shallow, narrow valleys of the loss landscapeloss landscape, n.: A way of picturing training as a journey across hilly terrain, where position stands for the model's settings and height for its error. Training searches for low valleys. and damps the noise of stochastic gradient descentstochastic gradient descent, n.: A faster version of gradient descent that estimates which way is downhill from a small random sample of the training data at each step, instead of checking the whole dataset.. Nesterov's accelerated variant evaluates the gradient at the look-ahead point the velocity is carrying the parameters toward; on smooth convex problems it attains the optimal worst-case rate O(1/t²), against O(1/t) for plain gradient descent. Momentum is built into most modern optimizersoptimizer, n.: The rule a training program uses to turn measured errors into actual changes to a model's internal numbers, deciding the direction and size of each adjustment.: AdamAdam, prop. n.: The most widely used method for training neural networks. It smooths the direction of updates over time and automatically sets a separate step size for each setting based on how large its recent changes have been.'s first-moment estimate is a momentum term, and MuonMuon, prop. n.: A newer training method for neural networks that tidies each update so it pushes evenly in all directions within a weight table. Several groups report that it trains language models with less computing than the standard method. orthogonalizes a momentum buffer.
Dropping momentum from 0.9 to 0.5 stopped the oscillation but slowed convergence.

v_{t+1} = \beta v_t + \nabla_{\theta} L(\theta_t), \qquad \theta_{t+1} = \theta_t - \eta \, v_{t+1}

Origin

Boris Polyak introduced the 'heavy ball' method in 1964, and Yurii Nesterov's accelerated gradient method followed in 1983. Rumelhart, Hinton and Williams used a momentum term in their 1986 backpropagation paper; Ilya Sutskever, James Martens, George Dahl and Geoffrey Hinton (2013) showed that well-tuned momentum could train deep and recurrent networks once thought to need second-order methods.

Usage note

Conventions differ: some libraries scale the gradient term by (1 − β), making the velocity an exponential moving average and changing the effective learning rate by a factor of 1/(1 − β).

Sources

Polyak, B. T. (1964). Some methods of speeding up the convergence of iteration methods. USSR Computational Mathematics and Mathematical Physics 4(5), 1–17.
Sutskever, I., Martens, J., Dahl, G. & Hinton, G. (2013). On the importance of initialization and momentum in deep learning. ICML 2013.

Mentioned in

gradient descentgradient descent, n.: A method for training models by repeatedly nudging every setting a small step in whichever direction most reduces the error, like a hiker who cannot see the valley but keeps stepping wherever the ground tilts down., AdamAdam, prop. n.: The most widely used method for training neural networks. It smooths the direction of updates over time and automatically sets a separate step size for each setting based on how large its recent changes have been., convergenceconvergence, n.: The point at which training settles down and further steps no longer meaningfully improve the model — and, in theory, a guarantee that a method will reach a solution, and how fast., MuonMuon, prop. n.: A newer training method for neural networks that tidies each update so it pushes evenly in all directions within a weight table. Several groups report that it trains language models with less computing than the standard method., optimizeroptimizer, n.: The rule a training program uses to turn measured errors into actual changes to a model's internal numbers, deciding the direction and size of each adjustment., RMSPropRMSProp, prop. n.: A training method that scales each setting's step by how large its recent changes have been, so jumpy settings move cautiously and quiet ones move faster., stochastic gradient descentstochastic gradient descent, n.: A faster version of gradient descent that estimates which way is downhill from a small random sample of the training data at each step, instead of checking the whole dataset., AdafactorAdafactor, prop. n.: A memory-saving cousin of the popular Adam training method. Instead of keeping extra statistics for every single weight, it keeps summaries for each row and column of a weight table., exponential moving average of weightsexponential moving average of weights, n.: A smoothed copy of a model kept during training, which each update nudges only slightly toward the latest weights. The calmer copy often performs better than the jittery original., LionLion, prop. n.: A training method discovered by a computer search over possible algorithms. It moves every setting by the same small amount at each step, choosing only the direction, and needs less memory than the popular Adam method., saddle pointsaddle point, n.: A spot on a surface that is a low point in one direction but a high point in another, like the middle of a horse's saddle or a mountain pass.

momentum

Origin

Usage note

Related

Sources

Mentioned in