gradient accumulation
In plain wordsA way to train with large batches on limited hardware: the computer processes several small batches one after another, adds up what each says, and only then makes a single update.
A technique that simulates a large batch sizebatch size, n.: How many training examples a model looks at before each adjustment of its settings. Bigger batches use hardware efficiently but bring diminishing returns; smaller ones are noisier but cheaper per step. by running the forward passforward pass, n.: The step where a neural network takes an input and runs it through its layers, one after another, to produce an output — its answer or prediction. and backward passbackward pass, n.: The step in training where the network works backward from its error, layer by layer, to calculate how each internal setting should change. on several smaller micro-batches in sequence, summing or averaging their gradients without applying them, and taking one optimizeroptimizer, n.: The rule a training program uses to turn measured errors into actual changes to a model's internal numbers, deciding the direction and size of each adjustment. step after k micro-batches. The update matches that of a batch k times larger — apart from layers that compute batch statistics, such as batch normalizationbatch normalization, n.: A layer that rescales the numbers flowing through a network so that, across each batch of training examples, they keep a steady average and spread. It makes deep networks train faster and more reliably. — at the cost of more wall-clock time per step rather than more memory. It is used to reach large effective batches on memory-limited accelerators, to keep the global batch fixed when the number of devices changes, and in pipeline parallelismpipeline parallelism, n.: Splitting a model's layers into consecutive stages on different chips, like stations on an assembly line, and streaming small batches of data through so that every station stays busy., where micro-batches flow through pipeline stages; in data-paralleldata parallelism, n.: Splitting the training data among many chips that each hold a full copy of the model; every chip learns from its own share, and they pool what they learned after each step so all copies stay identical. training, gradient synchronization can wait until the last micro-batch. Common pitfalls are forgetting to divide the loss by k and, for token-level losses, normalizing by the wrong token count when micro-batches contain different numbers of tokens.
With eight accumulation steps of 16 sequences we match the paper's batch of 128.
Origin
An engineering practice without a single inventor. It spread with large-batch neural machine translation and transformer training around 2017–2018 — for example in Myle Ott and colleagues' 2018 work on scaling translation models, which accumulated gradients over several sub-batches before each update — and became a standard option in training libraries.
Related
- batch sizebatch size, n.: How many training examples a model looks at before each adjustment of its settings. Bigger batches use hardware efficiently but bring diminishing returns; smaller ones are noisier but cheaper per step.
- mini-batchmini-batch, n.: A small random handful of training examples that a model learns from in a single step. Training works through many such handfuls instead of studying the whole dataset at once.
- gradient checkpointinggradient checkpointing, n.: A memory-saving trick: instead of storing every intermediate result needed for learning, the computer keeps only a few and recalculates the rest when needed, trading extra computing time for memory.
- pipeline parallelismpipeline parallelism, n.: Splitting a model's layers into consecutive stages on different chips, like stations on an assembly line, and streaming small batches of data through so that every station stays busy.
- data parallelismdata parallelism, n.: Splitting the training data among many chips that each hold a full copy of the model; every chip learns from its own share, and they pool what they learned after each step so all copies stay identical.
- training steptraining step, n.: One round of learning: the model makes predictions on a small batch of examples, measures its error and adjusts its settings once. Training runs consist of thousands to millions of steps.
- gradient checkpointinggradient checkpointing, n.: A memory-saving trick: instead of storing every intermediate result needed for learning, the computer keeps only a few and recalculates the rest when needed, trading extra computing time for memory.
Mentioned in
batch sizebatch size, n.: How many training examples a model looks at before each adjustment of its settings. Bigger batches use hardware efficiently but bring diminishing returns; smaller ones are noisier but cheaper per step., data parallelismdata parallelism, n.: Splitting the training data among many chips that each hold a full copy of the model; every chip learns from its own share, and they pool what they learned after each step so all copies stay identical., gradient checkpointinggradient checkpointing, n.: A memory-saving trick: instead of storing every intermediate result needed for learning, the computer keeps only a few and recalculates the rest when needed, trading extra computing time for memory., mini-batchmini-batch, n.: A small random handful of training examples that a model learns from in a single step. Training works through many such handfuls instead of studying the whole dataset at once., training looptraining loop, n.: The repeating cycle at the heart of training software: grab some examples, let the model guess, measure the error, adjust the model, and do it again — often millions of times., training steptraining step, n.: One round of learning: the model makes predictions on a small batch of examples, measures its error and adjusts its settings once. Training runs consist of thousands to millions of steps.