sequence parallelism
In plain wordsSplitting the long string of words a model is working on into chunks handled by different chips, which saves memory and makes it possible to train on very long documents.
- 1
Megatron sense
A complement to tensor parallelismtensor parallelism, n.: Splitting the individual giant tables of numbers inside each layer of a model across several chips, so that every chip does a slice of every calculation at the same time. that partitions activations along the sequence dimension in the parts of a transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it. layer that tensor parallelism leaves replicated, such as layer normalizationlayer normalization, n.: A layer that rescales the numbers inside a network, separately for each example, so they keep a steady average and spread. It, or a simpler variant, is a standard part of the models behind today's chatbots. and dropoutdropout, n.: A training trick that randomly switches off a fraction of a network's neurons at each step, so the network cannot lean too heavily on any one of them and learns sturdier patterns., so that no device stores full-sequence copies of those activations. Tensor parallelism's all-reduces are replaced by equivalent reduce-scatter and all-gather operations, so communication volume is unchanged while activation memory falls. Combined with selective activation recomputation, Vijay Korthikanti and colleagues (2022) reported about a fivefold reduction in activation memory and 54% model FLOPs utilizationmodel FLOPs utilization, n.: The share of a computer's top calculating speed that actually goes into useful AI training work. Real systems often reach only around a third to a half. when training a 530-billion-parameter model.
- 2
Long-sequence sense
Any scheme that distributes a single long sequence across devices for all layers, including attention, so that each device holds only a chunk of the tokens and attention requires exchanging keys and values between devices, for example around a ring. This sense is now usually called context parallelismcontext parallelism, n.: A way of training or running a model on extremely long inputs — whole books or codebases — by splitting the text across many chips and having them swap the information each part needs to 'see' the others..
Origin
Shenggui Li, Fuzhao Xue, Chaitanya Baranwal, Yongbin Li and Yang You introduced 'sequence parallelism' with ring self-attention for long sequences (arXiv, May 2021); Korthikanti and colleagues at NVIDIA reused the name for the Megatron-LM technique (May 2022), creating an ambiguity that 'context parallelism' later resolved.
Usage note
Check which sense is meant: in Megatron-LM, 'SP' is the activation-sharding complement to tensor parallelism; in DeepSpeed Ulysses and many papers, it means splitting whole sequences across devices.
Related
- context parallelismcontext parallelism, n.: A way of training or running a model on extremely long inputs — whole books or codebases — by splitting the text across many chips and having them swap the information each part needs to 'see' the others.
- tensor parallelismtensor parallelism, n.: Splitting the individual giant tables of numbers inside each layer of a model across several chips, so that every chip does a slice of every calculation at the same time.
- long contextlong context, n.: The ability of an AI model to work with very long inputs — whole books, codebases or hours of transcripts — in one go, and the techniques that make this possible.
- gradient checkpointinggradient checkpointing, n.: A memory-saving trick: instead of storing every intermediate result needed for learning, the computer keeps only a few and recalculates the rest when needed, trading extra computing time for memory.
- 3D parallelism3D parallelism, n.: Combining three ways of dividing up training work — by data, by layer stage and by slices within layers — so that thousands of chips can train one giant model efficiently.
- Megatron-LMMegatron-LM, prop. n.: NVIDIA's open-source software, first described in a 2019 research paper, for training very large language models across thousands of graphics chips by splitting each model in several ways at once.
Sources
- Li, S., Xue, F., Baranwal, C., Li, Y. & You, Y. (2021). Sequence Parallelism: Long Sequence Training from System Perspective. arXiv:2105.13120. arxiv.org/abs/2105.13120
- Korthikanti, V., Casper, J., Lym, S., McAfee, L., Andersch, M., Shoeybi, M. & Catanzaro, B. (2022). Reducing Activation Recomputation in Large Transformer Models. arXiv:2205.05198. arxiv.org/abs/2205.05198
Mentioned in
long contextlong context, n.: The ability of an AI model to work with very long inputs — whole books, codebases or hours of transcripts — in one go, and the techniques that make this possible., model parallelismmodel parallelism, n.: Splitting a single AI model that is too big for one chip into pieces spread over several chips, which must constantly pass intermediate results to each other as they work., context parallelismcontext parallelism, n.: A way of training or running a model on extremely long inputs — whole books or codebases — by splitting the text across many chips and having them swap the information each part needs to 'see' the others., Megatron-LMMegatron-LM, prop. n.: NVIDIA's open-source software, first described in a 2019 research paper, for training very large language models across thousands of graphics chips by splitting each model in several ways at once., tensor parallelismtensor parallelism, n.: Splitting the individual giant tables of numbers inside each layer of a model across several chips, so that every chip does a slice of every calculation at the same time.