Shampoo
In plain wordsA training method that adjusts each weight table's updates using statistics gathered along its rows and its columns, capturing how weights interact. Each step costs more computing, but fewer steps are needed.
A preconditioned optimizeroptimizer, n.: The rule a training program uses to turn measured errors into actual changes to a model's internal numbers, deciding the direction and size of each adjustment. that, for a weight matrix with gradient G, accumulates two small statistics matrices — L from GGᵀ and R from GᵀG — and updates the weights by L^(−1/4) G R^(−1/4): one preconditioner per tensor dimension instead of one enormous matrix over all parameters. This Kronecker-factored structure captures correlations between parameters, approximating full-matrix AdaGradAdaGrad, prop. n.: A training method that gives each of a model's settings its own step size, shrinking steps for settings that have already had many large updates and keeping them larger for rarely updated ones. at a memory cost of a few small matrices per layer. The inverse matrix roots are expensive, so practical versions recompute them only every few dozen to hundreds of steps, borrow ('graft') step sizes from a simpler optimizer, and spread the work across devices (Distributed Shampoo). SOAP (2024) showed Shampoo is closely related to running Adafactor or AdamAdam, prop. n.: The most widely used method for training neural networks. It smooths the direction of updates over time and automatically sets a separate step size for each setting based on how large its recent changes have been. in the eigenbasis of its preconditioner, and with accumulation switched off Shampoo's update equals the orthogonalized gradient computed by MuonMuon, prop. n.: A newer training method for neural networks that tidies each update so it pushes evenly in all directions within a weight table. Several groups report that it trains language models with less computing than the standard method.. A Distributed Shampoo submission won the external-tuning track of the first AlgoPerf training-algorithms competition (2024).
Origin
Introduced by Vineet Gupta, Tomer Koren and Yoram Singer of Google in 'Shampoo: Preconditioned Stochastic Tensor Optimization' (arXiv February 2018; ICML 2018); Rohan Anil and colleagues scaled it up in 2020. Its 2024 AlgoPerf win renewed interest in non-diagonal preconditioning; SOAP (Nikhil Vyas and colleagues, 2024) builds on it, and Muon is closely related to it.
Usage note
Although it uses curvature-like statistics, Shampoo needs only gradients and is often classed as a non-diagonal adaptive method rather than a true second-order methodsecond-order optimization, n.: Training methods that use not only the slope of the error but also how that slope is changing — its curvature — to choose better-sized steps. They can need far fewer steps, but each costs more..
Related
- second-order optimizationsecond-order optimization, n.: Training methods that use not only the slope of the error but also how that slope is changing — its curvature — to choose better-sized steps. They can need far fewer steps, but each costs more.
- AdaGradAdaGrad, prop. n.: A training method that gives each of a model's settings its own step size, shrinking steps for settings that have already had many large updates and keeping them larger for rarely updated ones.
- MuonMuon, prop. n.: A newer training method for neural networks that tidies each update so it pushes evenly in all directions within a weight table. Several groups report that it trains language models with less computing than the standard method.
- AdafactorAdafactor, prop. n.: A memory-saving cousin of the popular Adam training method. Instead of keeping extra statistics for every single weight, it keeps summaries for each row and column of a weight table.
- optimizeroptimizer, n.: The rule a training program uses to turn measured errors into actual changes to a model's internal numbers, deciding the direction and size of each adjustment.
- AdamAdam, prop. n.: The most widely used method for training neural networks. It smooths the direction of updates over time and automatically sets a separate step size for each setting based on how large its recent changes have been.
Sources
- Gupta, V., Koren, T. & Singer, Y. (2018). Shampoo: Preconditioned Stochastic Tensor Optimization. ICML 2018. arxiv.org/abs/1802.09568
- Vyas, N. et al. (2024). SOAP: Improving and Stabilizing Shampoo using Adam. arXiv:2409.11321. arxiv.org/abs/2409.11321
- Kasimbeg, P. et al. (2025). Accelerating Neural Network Training: An Analysis of the AlgoPerf Competition. ICLR 2025. arxiv.org/abs/2502.15015v1
Mentioned in
AdaGradAdaGrad, prop. n.: A training method that gives each of a model's settings its own step size, shrinking steps for settings that have already had many large updates and keeping them larger for rarely updated ones., MuonMuon, prop. n.: A newer training method for neural networks that tidies each update so it pushes evenly in all directions within a weight table. Several groups report that it trains language models with less computing than the standard method., optimizeroptimizer, n.: The rule a training program uses to turn measured errors into actual changes to a model's internal numbers, deciding the direction and size of each adjustment., AdafactorAdafactor, prop. n.: A memory-saving cousin of the popular Adam training method. Instead of keeping extra statistics for every single weight, it keeps summaries for each row and column of a weight table., AlgoPerfAlgoPerf, prop. n.: A public contest that measures which methods for training neural networks reach a set level of accuracy fastest on identical hardware, across several kinds of tasks, without being hand-tuned for each one., second-order optimizationsecond-order optimization, n.: Training methods that use not only the slope of the error but also how that slope is changing — its curvature — to choose better-sized steps. They can need far fewer steps, but each costs more.