3D parallelism

The combination of data parallelismdata parallelism, n.: Splitting the training data among many chips that each hold a full copy of the model; every chip learns from its own share, and they pool what they learned after each step so all copies stay identical., tensor parallelismtensor parallelism, n.: Splitting the individual giant tables of numbers inside each layer of a model across several chips, so that every chip does a slice of every calculation at the same time. and pipeline parallelismpipeline parallelism, n.: Splitting a model's layers into consecutive stages on different chips, like stations on an assembly line, and streaming small batches of data through so that every station stays busy. in one training job, with devices arranged as a logical grid whose axes correspond to the three strategies. The layout is matched to the hardware: tensor parallelism, which communicates most often, within a node over NVLinkNVLink, prop. n.: Nvidia's private high-speed connection between its AI chips, much faster than ordinary computer connections, letting dozens of chips share memory and act almost like one giant chip.; pipeline stages across nodes; and data parallelism, often sharded as in ZeROZeRO, prop. n.: A memory-saving technique for training big models on many chips: instead of every chip keeping its own complete copy of all the training bookkeeping, the chips divide it among themselves and share pieces only when needed. or FSDPfully sharded data parallel, n.: A feature of the popular PyTorch software that lets many chips train a model too big for any one of them, by spreading the model and its training bookkeeping evenly across all the chips., as the outermost dimension. Choosing the degree of each axis, with micro-batch size and activation recomputation, is a search over memory, communication and pipeline-bubble trade-offs. Adding context parallelismcontext parallelism, n.: A way of training or running a model on extremely long inputs — whole books or codebases — by splitting the text across many chips and having them swap the information each part needs to 'see' the others. gives '4D' parallelism — the layout Meta described for Llama 3 — and adding expert parallelismexpert parallelism, n.: A way of running a 'mixture of experts' model — one made of many specialist sub-networks — by giving each chip its own set of specialists and sending each piece of text to whichever chip holds the specialists it needs. for mixture of expertsmixture of experts, n.: A way to build very large AI models cheaply: the model contains many specialist sub-networks, and for each word only a few are switched on, picked by a small dispatcher network. models gives '5D'. With this approach Deepak Narayanan and colleagues trained a 1-trillion-parameter model on 3,072 GPUs at 502 petaFLOP/s, 52% of peak, in 2021.

Our 3D parallelism layout was TP 8 × PP 16 × DP 64 over 8,192 GPUs.