NCCL

NVIDIA's open-source library of GPU-optimized collective communicationcollective communication, n.: The standard patterns by which a whole group of chips exchanges data at once — for example, everyone adding up their numbers and all getting the total — which large training jobs use constantly. routines — all-reduceall-reduce, n.: A group operation in which every chip contributes a set of numbers, the sets are added together, and every chip ends up with the same total. It is how parallel training keeps model copies in sync., all-gather, reduce-scatter, broadcast, reduce, all-to-all and point-to-point send and receive — used by PyTorchPyTorch, prop. n.: The most widely used free software toolkit for building and training neural networks, especially in research; programmers write ordinary Python and it handles the calculus and the graphics chips., JAXJAX, prop. n.: A free Google library that lets researchers write mathematics in familiar Python and automatically get derivatives and very fast code for graphics chips and Google's AI chips., DeepSpeedDeepSpeed, prop. n.: Free software from Microsoft for training and running AI models too big for ordinary tools, spread over many chips; it is best known for a memory-saving technique called ZeRO., Megatron-LMMegatron-LM, prop. n.: NVIDIA's open-source software, first described in a 2019 research paper, for training very large language models across thousands of graphics chips by splitting each model in several ways at once. and most other frameworks as the communication backend for multi-GPU training and inference. NCCL detects the system topology (PCIe, NVLinkNVLink, prop. n.: Nvidia's private high-speed connection between its AI chips, much faster than ordinary computer connections, letting dozens of chips share memory and act almost like one giant chip. and NVSwitch within a node; InfiniBandInfiniBand, prop. n.: A very fast networking technology used to connect the thousands of servers in supercomputers and AI training clusters, designed so data moves with minimal delay. or RDMA-capable Ethernet between nodes), chooses ring, tree or other algorithms and protocols by message size, and runs communication as GPU kernels so that it can overlap with computation. Its performance and failure modes — hangs, timeouts and slow links — weigh heavily on the goodputgoodput, n.: The share of a big training job's time and computing that actually moves training forward, after subtracting time lost to crashes, restarts, waiting and repeated work. When serving users, it means the rate of requests answered fast enough. of large runs. AMD maintains a compatible fork, RCCL, for its GPUs.

The job hung in an NCCL all-reduce after one node's network card failed.