data parallelism
In plain wordsSplitting the training data among many chips that each hold a full copy of the model; every chip learns from its own share, and they pool what they learned after each step so all copies stay identical.
A parallelization strategy in which each of several workers holds a replica of the model and processes a different slice of each mini-batchmini-batch, n.: A small random handful of training examples that a model learns from in a single step. Training works through many such handfuls instead of studying the whole dataset at once.; after the backward passbackward pass, n.: The step in training where the network works backward from its error, layer by layer, to calculate how each internal setting should change., gradients are averaged across workers, typically with an all-reduceall-reduce, n.: A group operation in which every chip contributes a set of numbers, the sets are added together, and every chip ends up with the same total. It is how parallel training keeps model copies in sync., so that every replica applies the same update and stays synchronized. It is the simplest and most widely used form of distributed trainingdistributed training, n.: Training one AI model on many computer chips at once, splitting up the data or the model itself among them and keeping them in sync, so that jobs that would take a single chip thousands of years finish in weeks. — PyTorch's DistributedDataParallel (DDP) and Horovod are standard implementations — and communication is overlapped with the backward pass by reducing gradients in buckets as they become ready. Its limits are memory, since every worker stores the full model, gradients and optimizer state (addressed by sharded variants such as ZeROZeRO, prop. n.: A memory-saving technique for training big models on many chips: instead of every chip keeping its own complete copy of all the training bookkeeping, the chips divide it among themselves and share pieces only when needed. and FSDPfully sharded data parallel, n.: A feature of the popular PyTorch software that lets many chips train a model too big for any one of them, by spreading the model and its training bookkeeping evenly across all the chips.), and the critical batch sizecritical batch size, n.: The largest number of examples a model can usefully learn from in each training step. Below it, bigger batches let many chips work in parallel and finish sooner; above it, the extra chips are mostly wasted., since every added worker enlarges the global batch. Large runs combine it with model parallelismmodel parallelism, n.: Splitting a single AI model that is too big for one chip into pieces spread over several chips, which must constantly pass intermediate results to each other as they work., with data parallelism as the outermost dimension.
With data parallelism over 64 replicas, the global batch grew to 4 million tokens.
Origin
The term comes from parallel computing, notably W. Daniel Hillis and Guy L. Steele Jr.'s 'Data Parallel Algorithms' (Communications of the ACM, 1986). In deep learning it was used by Google's DistBelief (2012) and Alex Krizhevsky's 'One weird trick for parallelizing convolutional neural networks' (2014); ring all-reduce libraries such as Uber's Horovod (2017) made synchronous data parallelism the default.
Related
- distributed trainingdistributed training, n.: Training one AI model on many computer chips at once, splitting up the data or the model itself among them and keeping them in sync, so that jobs that would take a single chip thousands of years finish in weeks.
- all-reduceall-reduce, n.: A group operation in which every chip contributes a set of numbers, the sets are added together, and every chip ends up with the same total. It is how parallel training keeps model copies in sync.
- ZeROZeRO, prop. n.: A memory-saving technique for training big models on many chips: instead of every chip keeping its own complete copy of all the training bookkeeping, the chips divide it among themselves and share pieces only when needed.
- fully sharded data parallelfully sharded data parallel, n.: A feature of the popular PyTorch software that lets many chips train a model too big for any one of them, by spreading the model and its training bookkeeping evenly across all the chips.
- model parallelismmodel parallelism, n.: Splitting a single AI model that is too big for one chip into pieces spread over several chips, which must constantly pass intermediate results to each other as they work.
- critical batch sizecritical batch size, n.: The largest number of examples a model can usefully learn from in each training step. Below it, bigger batches let many chips work in parallel and finish sooner; above it, the extra chips are mostly wasted.
- gradient accumulationgradient accumulation, n.: A way to train with large batches on limited hardware: the computer processes several small batches one after another, adds up what each says, and only then makes a single update.
- model parallelismmodel parallelism, n.: Splitting a single AI model that is too big for one chip into pieces spread over several chips, which must constantly pass intermediate results to each other as they work.
Sources
- Hillis, W. D. & Steele, G. L., Jr. (1986). Data Parallel Algorithms. Communications of the ACM 29(12).
- Sergeev, A. & Del Balso, M. (2018). Horovod: fast and easy distributed deep learning in TensorFlow. arXiv:1802.05799. arxiv.org/abs/1802.05799
Mentioned in
backward passbackward pass, n.: The step in training where the network works backward from its error, layer by layer, to calculate how each internal setting should change., batch sizebatch size, n.: How many training examples a model looks at before each adjustment of its settings. Bigger batches use hardware efficiently but bring diminishing returns; smaller ones are noisier but cheaper per step., distributed trainingdistributed training, n.: Training one AI model on many computer chips at once, splitting up the data or the model itself among them and keeping them in sync, so that jobs that would take a single chip thousands of years finish in weeks., gradient accumulationgradient accumulation, n.: A way to train with large batches on limited hardware: the computer processes several small batches one after another, adds up what each says, and only then makes a single update., model parallelismmodel parallelism, n.: Splitting a single AI model that is too big for one chip into pieces spread over several chips, which must constantly pass intermediate results to each other as they work., 3D parallelism3D parallelism, n.: Combining three ways of dividing up training work — by data, by layer stage and by slices within layers — so that thousands of chips can train one giant model efficiently., all-reduceall-reduce, n.: A group operation in which every chip contributes a set of numbers, the sets are added together, and every chip ends up with the same total. It is how parallel training keeps model copies in sync., collective communicationcollective communication, n.: The standard patterns by which a whole group of chips exchanges data at once — for example, everyone adding up their numbers and all getting the total — which large training jobs use constantly., critical batch sizecritical batch size, n.: The largest number of examples a model can usefully learn from in each training step. Below it, bigger batches let many chips work in parallel and finish sooner; above it, the extra chips are mostly wasted., DiLoCoDiLoCo, prop. n.: A training method from Google DeepMind that lets groups of chips in different places train one language model while talking to each other hundreds of times less often than usual., fully sharded data parallelfully sharded data parallel, n.: A feature of the popular PyTorch software that lets many chips train a model too big for any one of them, by spreading the model and its training bookkeeping evenly across all the chips., local SGDlocal SGD, n.: A way to train on many machines that talk to each other only occasionally: each trains its own copy for a while, then they average their copies and continue, cutting the amount of communication needed., parameter serverparameter server, n.: An older design for training on many machines in which a few central computers hold the model and the others repeatedly fetch it, compute improvements on their share of data, and send them back., scale-up and scale-outscale-up and scale-out, phr.: Two ways to get more computing power: make each machine bigger and more tightly linked (scale up), or add more machines connected by a network (scale out)., stragglerstraggler, n.: One slow computer or chip in a large group working in lockstep. Because everyone must wait for the slowest member at each step, a single straggler can slow down an entire AI training job., ZeROZeRO, prop. n.: A memory-saving technique for training big models on many chips: instead of every chip keeping its own complete copy of all the training bookkeeping, the chips divide it among themselves and share pieces only when needed.