DeepSpeed
In plain wordsFree software from Microsoft for training and running AI models too big for ordinary tools, spread over many chips; it is best known for a memory-saving technique called ZeRO.
An open-source software library from Microsoft that extends PyTorchPyTorch, prop. n.: The most widely used free software toolkit for building and training neural networks, especially in research; programmers write ordinary Python and it handles the calculus and the graphics chips. with memory-saving and parallelization techniques for training and serving very large models across many GPUs. Its centerpiece is ZeROZeRO, prop. n.: A memory-saving technique for training big models on many chips: instead of every chip keeping its own complete copy of all the training bookkeeping, the chips divide it among themselves and share pieces only when needed., which shards optimizer state, gradients and parameters across data-parallel workers, with offloading to CPU memory and NVMe storage (ZeRO-Offload, ZeRO-Infinity). It also provides 3D parallelism3D parallelism, n.: Combining three ways of dividing up training work — by data, by layer stage and by slices within layers — so that thousands of chips can train one giant model efficiently. in combination with Megatron-style tensor parallelism — the pairing used for the 530-billion-parameter Megatron-Turing NLG — mixture-of-experts training and inference (DeepSpeed-MoE), long-sequence training (DeepSpeed Ulysses), mixed-precision trainingmixed-precision training, n.: Training a model with shorter, less exact numbers for most of the arithmetic and full-length numbers only where accuracy really matters, which saves a great deal of memory and lets modern chips train far faster. utilities and a reinforcement-learning-from-human-feedback pipeline (DeepSpeed-Chat, 2023). Applied through a configuration file around ordinary PyTorch code, it was a principal route by which ZeRO-style sharding reached open-source practice; PyTorch's native FSDPfully sharded data parallel, n.: A feature of the popular PyTorch software that lets many chips train a model too big for any one of them, by spreading the model and its training bookkeeping evenly across all the chips. later offered an alternative.
Switching on DeepSpeed's ZeRO-3 with CPU offload let the 30B model train on a single node.
Origin
Open-sourced by Microsoft in February 2020 with the first stage of ZeRO (proposed in an October 2019 paper), which had been used to train the 17-billion-parameter Turing-NLG model; developed by a team including Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase and Yuxiong He.
Related
- ZeROZeRO, prop. n.: A memory-saving technique for training big models on many chips: instead of every chip keeping its own complete copy of all the training bookkeeping, the chips divide it among themselves and share pieces only when needed.
- fully sharded data parallelfully sharded data parallel, n.: A feature of the popular PyTorch software that lets many chips train a model too big for any one of them, by spreading the model and its training bookkeeping evenly across all the chips.
- 3D parallelism3D parallelism, n.: Combining three ways of dividing up training work — by data, by layer stage and by slices within layers — so that thousands of chips can train one giant model efficiently.
- Megatron-LMMegatron-LM, prop. n.: NVIDIA's open-source software, first described in a 2019 research paper, for training very large language models across thousands of graphics chips by splitting each model in several ways at once.
- distributed trainingdistributed training, n.: Training one AI model on many computer chips at once, splitting up the data or the model itself among them and keeping them in sync, so that jobs that would take a single chip thousands of years finish in weeks.
- context parallelismcontext parallelism, n.: A way of training or running a model on extremely long inputs — whole books or codebases — by splitting the text across many chips and having them swap the information each part needs to 'see' the others.
Sources
- Microsoft Research (2020). ZeRO & DeepSpeed: New system optimizations enable training models with over 100 billion parameters. Microsoft Research Blog. microsoft.com/en-us/research/blog/zero-deepspeed-new-system-optimizations-enable-training-models-with-over-100-billion-parameters
Mentioned in
3D parallelism3D parallelism, n.: Combining three ways of dividing up training work — by data, by layer stage and by slices within layers — so that thousands of chips can train one giant model efficiently., expert parallelismexpert parallelism, n.: A way of running a 'mixture of experts' model — one made of many specialist sub-networks — by giving each chip its own set of specialists and sending each piece of text to whichever chip holds the specialists it needs., Megatron-LMMegatron-LM, prop. n.: NVIDIA's open-source software, first described in a 2019 research paper, for training very large language models across thousands of graphics chips by splitting each model in several ways at once., NCCLNCCL, prop. n.: NVIDIA's software library that lets its graphics chips exchange data with each other quickly — inside one computer and across networks — and that nearly all large-scale AI training on NVIDIA hardware relies on., PyTorch FoundationPyTorch Foundation, prop. n.: A nonprofit group, part of the Linux Foundation, that owns and steers the free PyTorch software for building AI so that no single company controls it. It now also looks after related projects., ZeROZeRO, prop. n.: A memory-saving technique for training big models on many chips: instead of every chip keeping its own complete copy of all the training bookkeeping, the chips divide it among themselves and share pieces only when needed.