Lion

An optimizeroptimizer, n.: The rule a training program uses to turn measured errors into actual changes to a model's internal numbers, deciding the direction and size of each adjustment. that keeps a single momentummomentum, n.: A training trick that lets each update keep part of the previous update's direction, like a ball rolling downhill that builds up speed and coasts over small bumps. buffer and updates each parameter by the sign of an interpolation between that momentum and the current gradient, so every coordinate moves by exactly the learning ratelearning rate, n.: How big a step a model takes each time it adjusts its settings during training. Too large and learning goes haywire; too small and it crawls. in one direction or the other; weight decayweight decay, n.: A training rule that shrinks every weight slightly at each step, like a small tax on large numbers. It keeps models simpler and usually helps them work better on new data. is applied in decoupled form, as in AdamWAdamW, prop. n.: A corrected version of the popular Adam training method that applies its steady pull toward smaller weights separately from the main update. It is the standard choice for training large language models.. Keeping one state tensor instead of Adam's two saves memory, and sign updates are cheap to compute and to communicate. The sign operation gives its updates a larger norm than Adam's, so Lion is run with a learning rate several times smaller and a correspondingly larger weight decay. Its authors reported gains across language modeling, diffusion models, vision–language contrastive training and image classification; independent comparisons have generally found its advantage over well-tuned AdamW modest and task-dependent. It is a prominent example of an optimizer found by automated program search rather than derived by hand, and belongs to the sign-based family that includes signSGD and Rprop.