Skip to content
Last Dictionary
Plane
Training & Post-trainingHardware & Compute

pipeline parallelism

nounAlso: PP, pipeline model parallelism, inter-layer model parallelism, pipeline bubble, 1F1B

In plain wordsSplitting a model's layers into consecutive stages on different chips, like stations on an assembly line, and streaming small batches of data through so that every station stays busy.

  1. A form of model parallelismmodel parallelism, n.: Splitting a single AI model that is too big for one chip into pieces spread over several chips, which must constantly pass intermediate results to each other as they work. that partitions a network's layers into sequential stages on different devices and splits each mini-batchmini-batch, n.: A small random handful of training examples that a model learns from in a single step. Training works through many such handfuls instead of studying the whole dataset at once. into micro-batches that flow through the stages like an assembly line, so that different stages work on different micro-batches at once. Only activations at stage boundaries, and their gradients, are communicated, point to point, which suits slower links between nodes. Its inefficiency is the pipeline bubble: idle time while the pipeline fills and drains, a fraction (p − 1)/(m + p − 1) of each step for p stages and m micro-batches under the simple GPipe schedule. Later schedules reduce the bubble or memory: one-forward-one-backward (1F1B) interleaving, Megatron-LM's interleaved virtual stages, 'zero-bubble' schedules that split the backward pass, and DeepSeek's DualPipe, which overlaps computation with communication. Asynchronous variants such as PipeDream avoid flushing the pipeline but must manage stale weights.

    With 16 pipeline stages and 64 micro-batches, the bubble still took about 19% of each step.

bubble fraction=p−1m+p−1\text{bubble fraction} = \frac{p - 1}{m + p - 1}

Origin

Pipelining is old in computer architecture. For neural-network training it was introduced at scale by PipeDream (Aaron Harlap, Deepak Narayanan and colleagues, June 2018) and Google's GPipe (Yanping Huang and colleagues, November 2018); Narayanan and colleagues' interleaved schedule (2021) became standard in Megatron-LM.

Related

See also
  • model parallelismmodel parallelism, n.: Splitting a single AI model that is too big for one chip into pieces spread over several chips, which must constantly pass intermediate results to each other as they work.
  • tensor parallelismtensor parallelism, n.: Splitting the individual giant tables of numbers inside each layer of a model across several chips, so that every chip does a slice of every calculation at the same time.
  • 3D parallelism3D parallelism, n.: Combining three ways of dividing up training work — by data, by layer stage and by slices within layers — so that thousands of chips can train one giant model efficiently.
  • Megatron-LMMegatron-LM, prop. n.: NVIDIA's open-source software, first described in a 2019 research paper, for training very large language models across thousands of graphics chips by splitting each model in several ways at once.
  • mini-batchmini-batch, n.: A small random handful of training examples that a model learns from in a single step. Training works through many such handfuls instead of studying the whole dataset at once.
  • distributed trainingdistributed training, n.: Training one AI model on many computer chips at once, splitting up the data or the model itself among them and keeping them in sync, so that jobs that would take a single chip thousands of years finish in weeks.

Sources

  1. Huang, Y., Cheng, Y., Bapna, A., Firat, O., Chen, M. X. et al. (2018). GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism. arXiv:1811.06965. arxiv.org/abs/1811.06965
  2. Harlap, A., Narayanan, D., Phanishayee, A. et al. (2018). PipeDream: Fast and Efficient Pipeline Parallel DNN Training. arXiv:1806.03377. arxiv.org/abs/1806.03377
  3. Narayanan, D., Shoeybi, M., Casper, J. et al. (2021). Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM. arXiv:2104.04473. arxiv.org/abs/2104.04473

Mentioned in

distributed trainingdistributed training, n.: Training one AI model on many computer chips at once, splitting up the data or the model itself among them and keeping them in sync, so that jobs that would take a single chip thousands of years finish in weeks., gradient accumulationgradient accumulation, n.: A way to train with large batches on limited hardware: the computer processes several small batches one after another, adds up what each says, and only then makes a single update., machine learning pipelinemachine learning pipeline, n.: An automated chain of steps that takes raw data, prepares it, trains and tests a model and hands it over for use, so the whole process can be rerun the same way., model parallelismmodel parallelism, n.: Splitting a single AI model that is too big for one chip into pieces spread over several chips, which must constantly pass intermediate results to each other as they work., vLLMvLLM, prop. n.: A free, open-source program for running large language models on servers efficiently, so one machine can answer many more users at once. Many companies use it to host AI models., 3D parallelism3D parallelism, n.: Combining three ways of dividing up training work — by data, by layer stage and by slices within layers — so that thousands of chips can train one giant model efficiently., fully sharded data parallelfully sharded data parallel, n.: A feature of the popular PyTorch software that lets many chips train a model too big for any one of them, by spreading the model and its training bookkeeping evenly across all the chips., Megatron-LMMegatron-LM, prop. n.: NVIDIA's open-source software, first described in a 2019 research paper, for training very large language models across thousands of graphics chips by splitting each model in several ways at once., scale-up and scale-outscale-up and scale-out, phr.: Two ways to get more computing power: make each machine bigger and more tightly linked (scale up), or add more machines connected by a network (scale out)., stragglerstraggler, n.: One slow computer or chip in a large group working in lockstep. Because everyone must wait for the slowest member at each step, a single straggler can slow down an entire AI training job., tensor parallelismtensor parallelism, n.: Splitting the individual giant tables of numbers inside each layer of a model across several chips, so that every chip does a slice of every calculation at the same time.