numerical stability

The property of an algorithm whose output is not unduly corrupted by the rounding errors of finite-precision arithmetic; an unstable algorithm can amplify small errors into large ones even when the underlying problem is well-posed. It interacts with the problem's conditioning: the condition number measures how sensitive the exact answer is to small input changes, whereas stability concerns the method. Deep learning is exposed because it runs billions of operations in low-precision floating-point formatsfloating-point format, n.: A way of storing numbers in a computer like scientific notation — a few digits plus a power of two — trading accuracy for range. AI uses ever-smaller formats to compute faster. such as FP16FP16, abbr.: A 16-bit way of storing numbers that uses half the memory of the standard 32-bit format. It is fast for AI but can't represent very large or very tiny values., bfloat16bfloat16, n.: A 16-bit number format invented at Google for AI. It keeps the huge range of the standard 32-bit format but drops precision, which neural networks can usually tolerate. and FP8FP8, abbr.: An 8-bit number format that uses a quarter of the memory of the standard 32-bit format, letting new AI chips train and run models roughly twice as fast as with 16 bits.. Standard remedies include the log-sum-exp trick (subtracting the maximum before exponentiating in softmaxsoftmax, n.: A formula that turns a list of raw scores into probabilities — all positive and adding up to one — with the biggest share going to the highest score. and cross-entropycross-entropy, n.: A score of how surprised a model's predictions are by what actually happens; lower is better. Language models are trained mainly by driving this number down on real text.), computing losses from logitslogit, n.: A raw score that a model gives each possible answer before the scores are turned into probabilities. In statistics, the 'log-odds': a probability rewritten on a scale running from minus to plus infinity. and log-probabilities rather than probabilities, small epsilons in divisions and normalizations, loss scaling in mixed-precision trainingmixed-precision training, n.: Training a model with shorter, less exact numbers for most of the arithmetic and full-length numbers only where accuracy really matters, which saves a great deal of memory and lets modern chips train far faster., gradient clippinggradient clipping, n.: A safety limit on how big a single training update can be. If the computed adjustment is unusually large, it is scaled down so that one bad batch cannot wreck the model. and careful weight initializationweight initialization, n.: Choosing the random starting values of a neural network's internal numbers before training. Good starting values keep signals from fading away or blowing up as they pass through many layers.. Instability shows up as NaN or infinite values, loss spikesloss spike, n.: A sudden jump in a model's error partway through training. Sometimes the model recovers on its own; sometimes the run falls apart and engineers must rewind to an earlier saved copy. or divergence.

Compute the loss from logits with log_softmax; taking the log of softmax outputs is not numerically stable.