silent data corruption
In plain wordsWhen computer hardware quietly produces wrong numbers without crashing or raising any alarm. In huge AI training runs on thousands of chips, such hidden errors can damage a model before anyone notices.
An error in which hardware returns an incorrect result, in computation, memory or data transfer, without the error being detected or reported: no crash, no exception and no error-correcting-code (ECC) alert. Causes include manufacturing defects and wear-out in processor logic, marginal voltage, frequency or temperature, and occasionally radiation-induced bit flips that evade protection; a defective unit may err only on particular instructions, data patterns or operating conditions, which makes it hard to find. In large-scale training, where thousands of GPUsGPU, abbr.: A computer chip first designed to draw video-game graphics that turned out to be ideal for AI, because it can do thousands of simple calculations at the same time. or TPUsTPU, abbr.: Google's own family of chips built specifically for AI, used to train and run its Gemini models and rented to other companies through its cloud. run for months, even rare per-chip error rates add up to recurring events, and a corrupted gradient can spread through all-reduceall-reduce, n.: A group operation in which every chip contributes a set of numbers, the sets are added together, and every chip ends up with the same total. It is how parallel training keeps model copies in sync. into every replica's weights, appearing as loss spikesloss spike, n.: A sudden jump in a model's error partway through training. Sometimes the model recovers on its own; sometimes the run falls apart and engineers must rewind to an earlier saved copy., NaNs or a subtle loss of quality. Defenses include fleet screening and burn-in tests, replaying or duplicating computation and comparing results, checking that data-parallel replicas stay bit-identical, monitoring gradient norms, frequent checkpointscheckpoint, n.: A saved snapshot of a model partway through or at the end of training, like a save point in a video game: training can resume from it after a crash, and finished snapshots are what people download and run. for rollback, and quarantining suspect hosts.
The loss spike vanished when we replayed the step on another node, so we pulled the first host for silent-data-corruption testing.
Origin
The phrase was established in storage and high-performance computing by the 2000s, for corruption that slips past disk and memory checks. It gained prominence for processors in 2021, when engineers at Facebook, now Meta (Harish Dattatraya Dixit and colleagues), and Google (Peter Hochschild and colleagues, who called the culprits 'mercurial cores') reported defective CPUs silently miscomputing at fleet scale. Frontier-model technical reports since 2023 treat it as a recurring training hazard.
Usage note
Distinct from detected or corrected errors, which hardware reports, and from software nondeterminismnondeterminism, n.: The fact that an AI model can give different answers to exactly the same question, partly because it deliberately chooses words with some randomness and partly because of tiny computing differences., which yields slightly different but valid results. Meta attributed 6 of the 419 unexpected interruptions (466 in all) in a 54-day period of Llama 3 405B pretraining, on up to 16,384 H100 GPUs, to silent data corruption: few, but among the hardest to diagnose.
Related
- GPUGPU, abbr.: A computer chip first designed to draw video-game graphics that turned out to be ideal for AI, because it can do thousands of simple calculations at the same time.
- checkpointcheckpoint, n.: A saved snapshot of a model partway through or at the end of training, like a save point in a video game: training can resume from it after a crash, and finished snapshots are what people download and run.
- loss spikeloss spike, n.: A sudden jump in a model's error partway through training. Sometimes the model recovers on its own; sometimes the run falls apart and engineers must rewind to an earlier saved copy.
- training instabilitytraining instability, n.: The tendency of very large training runs to go wrong — the error suddenly jumping or blowing up — which can waste weeks of expensive computing unless engineers prevent or repair it.
- stragglerstraggler, n.: One slow computer or chip in a large group working in lockstep. Because everyone must wait for the slowest member at each step, a single straggler can slow down an entire AI training job.
- goodputgoodput, n.: The share of a big training job's time and computing that actually moves training forward, after subtracting time lost to crashes, restarts, waiting and repeated work. When serving users, it means the rate of requests answered fast enough.
- all-reduceall-reduce, n.: A group operation in which every chip contributes a set of numbers, the sets are added together, and every chip ends up with the same total. It is how parallel training keeps model copies in sync.
- nondeterminismnondeterminism, n.: The fact that an AI model can give different answers to exactly the same question, partly because it deliberately chooses words with some randomness and partly because of tiny computing differences.
Sources
- Dixit, H. D. et al. (2021). Silent Data Corruptions at Scale. arXiv preprint.
- Hochschild, P. H. et al. (2021). Cores that Don't Count. Proc. HotOS 2021.
Mentioned in
stragglerstraggler, n.: One slow computer or chip in a large group working in lockstep. Because everyone must wait for the slowest member at each step, a single straggler can slow down an entire AI training job.