AdamW

A variant of AdamAdam, prop. n.: The most widely used method for training neural networks. It smooths the direction of updates over time and automatically sets a separate step size for each setting based on how large its recent changes have been. that applies weight decayweight decay, n.: A training rule that shrinks every weight slightly at each step, like a small tax on large numbers. It keeps models simpler and usually helps them work better on new data. directly to the parameters — shrinking each by a small fraction at every step (in most libraries, the learning rate times the decay coefficient) — instead of adding an L2 penalty to the loss. The difference matters only for adaptive methods: Adam divides an L2 penalty's gradient by its second-moment estimate, so weights with large historical gradients are shrunk less than intended, whereas under plain stochastic gradient descentstochastic gradient descent, n.: A faster version of gradient descent that estimates which way is downhill from a small random sample of the training data at each step, instead of checking the whole dataset. the two formulations coincide. Decoupling restores uniform shrinkage, and its authors found that it made good decay values much less dependent on the learning ratelearning rate, n.: How big a step a model takes each time it adjusts its settings during training. Too large and learning goes haywire; too small and it crawls.. AdamW generally generalizes better than Adam with L2 and became the default optimizer for transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it. pretrainingpretraining, n.: The first and usually the most expensive stage of building a modern AI model, in which it learns general patterns from enormous amounts of raw text, images or code before being specialized for particular uses. and fine-tuning; a widely copied large-language-model recipe (Meta's LLaMA, 2023) uses β₁ = 0.9, β₂ = 0.95, decay 0.1, gradient clippinggradient clipping, n.: A safety limit on how big a single training update can be. If the computed adjustment is unusually large, it is scaled down so that one bad batch cannot wreck the model. at 1.0, warmup and cosine decay. As of 2026 it remains the baseline against which new optimizers such as MuonMuon, prop. n.: A newer training method for neural networks that tidies each update so it pushes evenly in all directions within a weight table. Several groups report that it trains language models with less computing than the standard method. are measured, and it still trains the embeddings and normalization parameters of Muon-trained models.

We fine-tuned with AdamW at 2e-5 and weight decay 0.01, excluding biases and norms from decay.