convergence

Optimization theory

The property that the iterates of an optimization algorithm approach a solution — a minimizer, a stationary point where the gradient vanishes, or a target loss value — as the number of steps grows, together with the rate at which they do so. For gradient descentgradient descent, n.: A method for training models by repeatedly nudging every setting a small step in whichever direction most reduces the error, like a hiker who cannot see the valley but keeps stepping wherever the ground tilts down. on smooth convex functions the error falls as O(1/t), with Nesterov momentummomentum, n.: A training trick that lets each update keep part of the previous update's direction, like a ball rolling downhill that builds up speed and coasts over small bumps. O(1/t²), and geometrically under strong convexity; stochastic gradient descentstochastic gradient descent, n.: A faster version of gradient descent that estimates which way is downhill from a small random sample of the training data at each step, instead of checking the whole dataset. on convex problems achieves O(1/√t) with decaying step sizes. For the non-convex losses of neural networks, guarantees usually cover only reaching an approximately stationary point (for SGD, gradient norm below ε within on the order of ε⁻⁴ steps), not a global minimum, although sufficiently wide networks provably reach zero training loss in some regimes (see neural tangent kernelneural tangent kernel, n.: A mathematical tool showing that extremely wide neural networks learn like a simpler, well-understood method, because their internal settings barely move during training. It explains some, but not all, of deep learning's behavior.).

Practice

The informal state in which a training runtraining run, n.: One complete attempt at training a model with a fixed recipe, from start to finish. For the largest AI models a single run can occupy tens of thousands of chips for months.'s loss curveloss curve, n.: A graph of a model's error as training goes on. A healthy curve drops steeply and then keeps sloping gently downward; bumps, jumps or flat stretches can warn engineers that something is wrong. has flattened, so further steps yield negligible improvement in training or validation loss; 'trained to convergence' contrasts with runs stopped early for budget reasons. Apparent convergence can mislead: plateaus may precede sudden drops, grokkinggrokking, n.: When a model that has merely memorized its training examples suddenly, long afterward, figures out the underlying rule and starts getting new examples right. can arrive long after training accuracy saturates, and a learning rate schedulelearning rate schedule, n.: A plan for changing how big a model's learning steps are during training — usually starting cautiously, then taking big steps, then smaller and smaller ones as it settles in. that decays to zero flattens the curve whether or not more learning was possible.

The 7B model had not converged after 2 trillion tokens; loss was still falling.