loss spike
In plain wordsA sudden jump in a model's error partway through training. Sometimes the model recovers on its own; sometimes the run falls apart and engineers must rewind to an earlier saved copy.
An abrupt, transient increase in training loss during a run, typically lasting tens to hundreds of steps, after which the loss either recovers or diverges entirely. Spikes become more frequent at large scale and high learning rateslearning rate, n.: How big a step a model takes each time it adjusts its settings during training. Too large and learning goes haywire; too small and it crawls.. Proposed causes include rapid growth of attention or output logitslogit, n.: A raw score that a model gives each possible answer before the scores are turned into probabilities. In statistics, the 'log-odds': a probability rewritten on a scale running from minus to plus infinity., interactions between particular data batches and the current parameter state, Adam's second-moment estimates shrinking during long stretches of tiny gradients so that a sudden larger gradient produces an outsized update, and overflow in low-precision arithmetic. Mitigations include lower peak learning rates and longer warmuplearning rate warmup, n.: Starting training with tiny learning steps and gradually increasing them over the first stretch of training, so the model does not lurch wildly before it has found its footing., gradient clippinggradient clipping, n.: A safety limit on how big a single training update can be. If the computed adjustment is unusually large, it is scaled down so that one bad batch cannot wreck the model., QK-normQK-norm, n.: A small change to how a language model pays attention — rescaling two sets of internal numbers before comparing them — that prevents a common way large models' training goes haywire. and z-lossz-loss, n.: A small extra penalty added during training that stops the raw scores a language model produces before choosing a word from growing huge, which helps keep very large training runs from blowing up., adjusted optimizer settings, removing or reordering offending data, and the standard recovery: restarting from a checkpointcheckpoint, n.: A saved snapshot of a model partway through or at the end of training, like a save point in a video game: training can resume from it after a crash, and finished snapshots are what people download and run. taken before the spike while skipping the batches that preceded it. Google's 540-billion-parameter PaLM run saw about 20 spikes despite gradient clipping.
We rolled back 100 steps and skipped 300 batches, and the loss spike did not recur.
Origin
Informal practitioner term that became prominent with reports on large language-model runs in 2022 — Meta's OPT-175B logbook and Google's PaLM, which documented restarts from about 100 steps before a spike that skipped roughly 200–500 batches — and with later analyses of Adam-related causes (Igor Molybog and colleagues, 2023).
Related
- training instabilitytraining instability, n.: The tendency of very large training runs to go wrong — the error suddenly jumping or blowing up — which can waste weeks of expensive computing unless engineers prevent or repair it.
- loss curveloss curve, n.: A graph of a model's error as training goes on. A healthy curve drops steeply and then keeps sloping gently downward; bumps, jumps or flat stretches can warn engineers that something is wrong.
- z-lossz-loss, n.: A small extra penalty added during training that stops the raw scores a language model produces before choosing a word from growing huge, which helps keep very large training runs from blowing up.
- QK-normQK-norm, n.: A small change to how a language model pays attention — rescaling two sets of internal numbers before comparing them — that prevents a common way large models' training goes haywire.
- gradient clippinggradient clipping, n.: A safety limit on how big a single training update can be. If the computed adjustment is unusually large, it is scaled down so that one bad batch cannot wreck the model.
- checkpointcheckpoint, n.: A saved snapshot of a model partway through or at the end of training, like a save point in a video game: training can resume from it after a crash, and finished snapshots are what people download and run.
Sources
- Chowdhery, A. et al. (2022). PaLM: Scaling Language Modeling with Pathways. arXiv:2204.02311.
- Molybog, I. et al. (2023). A Theory on Adam Instability in Large-Scale Machine Learning. arXiv preprint.
Mentioned in
checkpointcheckpoint, n.: A saved snapshot of a model partway through or at the end of training, like a save point in a video game: training can resume from it after a crash, and finished snapshots are what people download and run., exploding gradient problemexploding gradient problem, n.: A training failure in which the learning signal grows out of control as it passes back through a network, so the weights jump wildly and training breaks down., gradient clippinggradient clipping, n.: A safety limit on how big a single training update can be. If the computed adjustment is unusually large, it is scaled down so that one bad batch cannot wreck the model., learning rate warmuplearning rate warmup, n.: Starting training with tiny learning steps and gradually increasing them over the first stretch of training, so the model does not lurch wildly before it has found its footing., loss curveloss curve, n.: A graph of a model's error as training goes on. A healthy curve drops steeply and then keeps sloping gently downward; bumps, jumps or flat stretches can warn engineers that something is wrong., numerical stabilitynumerical stability, n.: Whether a computer calculation stays accurate despite the tiny rounding errors in every step, instead of blowing up into nonsense such as infinities., training instabilitytraining instability, n.: The tendency of very large training runs to go wrong — the error suddenly jumping or blowing up — which can waste weeks of expensive computing unless engineers prevent or repair it., training runtraining run, n.: One complete attempt at training a model with a fixed recipe, from start to finish. For the largest AI models a single run can occupy tens of thousands of chips for months., QK-normQK-norm, n.: A small change to how a language model pays attention — rescaling two sets of internal numbers before comparing them — that prevents a common way large models' training goes haywire., silent data corruptionsilent data corruption, n.: When computer hardware quietly produces wrong numbers without crashing or raising any alarm. In huge AI training runs on thousands of chips, such hidden errors can damage a model before anyone notices., z-lossz-loss, n.: A small extra penalty added during training that stops the raw scores a language model produces before choosing a word from growing huge, which helps keep very large training runs from blowing up.