Muon
In plain wordsA newer training method for neural networks that tidies each update so it pushes evenly in all directions within a weight table. Several groups report that it trains language models with less computing than the standard method.
An optimizeroptimizer, n.: The rule a training program uses to turn measured errors into actual changes to a model's internal numbers, deciding the direction and size of each adjustment. that treats each hidden-layer weight matrix as a whole rather than as a list of independent numbers: it computes the usual momentummomentum, n.: A training trick that lets each update keep part of the previous update's direction, like a ball rolling downhill that builds up speed and coasts over small bumps. update of stochastic gradient descentstochastic gradient descent, n.: A faster version of gradient descent that estimates which way is downhill from a small random sample of the training data at each step, instead of checking the whole dataset. (Nesterov-style by default), then snaps that update to a nearby approximately orthogonal matrix, keeping its singular vectors but setting every singular value close to one. A few low-precision Newton–Schulz iterations (five in the reference implementation) do the snapping instead of an exact singular value decompositionsingular value decomposition, n.: A way of rewriting any grid of numbers as three simple moves in turn — turn, stretch along perpendicular axes, turn again — which reveals the directions that matter most in the data.. Mathematically this is steepest descent under the spectral norm; Keller Jordan's rationale is that raw momentum updates are dominated by a few large singular directions, and equalizing them boosts rarer directions that still matter for learning. Parameters that are not hidden-layer matrices (embeddings, output heads, biases and normalization gains) are left to AdamWAdamW, prop. n.: A corrected version of the popular Adam training method that applies its steady pull toward smaller weights separately from the main update. It is the standard choice for training large language models.. Scaling Muon to large models required adding weight decayweight decay, n.: A training rule that shrinks every weight slightly at each step, like a small tax on large numbers. It keeps models simpler and usually helps them work better on new data., matching update magnitudes to AdamW's, and — in Moonshot AI's trillion-parameter Kimi K2 (2025) — 'QK-clip', which rescales query and key weights to stop attention logits exploding; the combination is called MuonClip.
Muon reached the target loss in about 70% of the AdamW baseline's tokens, but only after we tuned its weight decay.
Origin
Developed by Keller Jordan in October 2024 for the NanoGPT and CIFAR-10 'speedrunning' competitions, building on Jeremy Bernstein and Laker Newhouse's analysis of optimizers as steepest descent under norms; it set a NanoGPT speedrun record on 15 October 2024 and was written up that December. Moonshot AI's 'Muon is Scalable for LLM Training' (February 2025) reported about twice AdamW's compute efficiency, and Kimi K2 (July 2025) used MuonClip for 15.5 trillion pretraining tokens.
Usage note
Reported speedups vary with scale, tuning and baseline quality: an independent 2025 benchmark by Kaiyue Wen and colleagues found the advantage of matrix-based optimizers such as Muon over well-tuned AdamW shrinking from 1.4× at 0.1 billion parameters to 1.1× at 1.2 billion. How far it holds for the largest models was still debated as of October 2026.
Related
- optimizeroptimizer, n.: The rule a training program uses to turn measured errors into actual changes to a model's internal numbers, deciding the direction and size of each adjustment.
- AdamWAdamW, prop. n.: A corrected version of the popular Adam training method that applies its steady pull toward smaller weights separately from the main update. It is the standard choice for training large language models.
- ShampooShampoo, prop. n.: A training method that adjusts each weight table's updates using statistics gathered along its rows and its columns, capturing how weights interact. Each step costs more computing, but fewer steps are needed.
- momentummomentum, n.: A training trick that lets each update keep part of the previous update's direction, like a ball rolling downhill that builds up speed and coasts over small bumps.
- second-order optimizationsecond-order optimization, n.: Training methods that use not only the slope of the error but also how that slope is changing — its curvature — to choose better-sized steps. They can need far fewer steps, but each costs more.
- singular value decompositionsingular value decomposition, n.: A way of rewriting any grid of numbers as three simple moves in turn — turn, stretch along perpendicular axes, turn again — which reveals the directions that matter most in the data.
Sources
- Jordan, K. et al. (2024). Muon: An optimizer for hidden layers in neural networks. Blog post, 8 December 2024. kellerjordan.github.io/posts/muon
- Liu, J. et al. (2025). Muon is Scalable for LLM Training. arXiv:2502.16982. arxiv.org/html/2502.16982v1
- Kimi Team (2025). Kimi K2: Open Agentic Intelligence. arXiv:2507.20534. arxiv.org/html/2507.20534v2
Mentioned in
AdamAdam, prop. n.: The most widely used method for training neural networks. It smooths the direction of updates over time and automatically sets a separate step size for each setting based on how large its recent changes have been., AdamWAdamW, prop. n.: A corrected version of the popular Adam training method that applies its steady pull toward smaller weights separately from the main update. It is the standard choice for training large language models., gradient clippinggradient clipping, n.: A safety limit on how big a single training update can be. If the computed adjustment is unusually large, it is scaled down so that one bad batch cannot wreck the model., momentummomentum, n.: A training trick that lets each update keep part of the previous update's direction, like a ball rolling downhill that builds up speed and coasts over small bumps., Moonshot AIMoonshot AI, prop. n.: A Beijing artificial intelligence company, founded in 2023, that makes the Kimi chatbot and very large models that anyone can download., optimizeroptimizer, n.: The rule a training program uses to turn measured errors into actual changes to a model's internal numbers, deciding the direction and size of each adjustment., singular value decompositionsingular value decomposition, n.: A way of rewriting any grid of numbers as three simple moves in turn — turn, stretch along perpendicular axes, turn again — which reveals the directions that matter most in the data., orthogonalityorthogonality, n.: The mathematical version of being at right angles. Two directions are orthogonal when moving along one tells you nothing about the other, like north and east on a map., second-order optimizationsecond-order optimization, n.: Training methods that use not only the slope of the error but also how that slope is changing — its curvature — to choose better-sized steps. They can need far fewer steps, but each costs more., ShampooShampoo, prop. n.: A training method that adjusts each weight table's updates using statistics gathered along its rows and its columns, capturing how weights interact. Each step costs more computing, but fewer steps are needed.