interconnect
In plain wordsThe wiring and networking that connect chips to each other — inside a server, across a data center, or within a single chip — so they can share data and work as one.
- 1
AI systems
The communication links and switches that join processors and memory into a larger machine, characterized by bandwidth, latency, topology and scale. AI systems use a hierarchy: die-to-die links inside a package (such as the 10 TB/s link between the two dies of an Nvidia Blackwell GPU, or chipletchiplet, n.: One of several smaller chips that are made separately and then joined inside a single package to work as one processor, which is cheaper than making one giant chip and lets makers reuse parts. standards such as UCIe); a scale-upscale-up and scale-out, phr.: Two ways to get more computing power: make each machine bigger and more tightly linked (scale up), or add more machines connected by a network (scale out). domain joining tens to thousands of accelerators with memory-like semantics (NVLinkNVLink, prop. n.: Nvidia's private high-speed connection between its AI chips, much faster than ordinary computer connections, letting dozens of chips share memory and act almost like one giant chip., AMD's Infinity Fabric, Google's TPU inter-chip interconnect, the open UALink standard); and a scale-out network linking many servers (InfiniBandInfiniBand, prop. n.: A very fast networking technology used to connect the thousands of servers in supercomputers and AI training clusters, designed so data moves with minimal delay., or Ethernet with RDMARDMA, abbr.: A way for one computer to read or write another computer's memory directly over the network, skipping the usual software middlemen, so data moves faster with less delay., including the Ultra Ethernet specification). Because distributed trainingdistributed training, n.: Training one AI model on many computer chips at once, splitting up the data or the model itself among them and keeping them in sync, so that jobs that would take a single chip thousands of years finish in weeks. depends on frequent collective operationscollective communication, n.: The standard patterns by which a whole group of chips exchanges data at once — for example, everyone adding up their numbers and all getting the total — which large training jobs use constantly. such as all-reduceall-reduce, n.: A group operation in which every chip contributes a set of numbers, the sets are added together, and every chip ends up with the same total. It is how parallel training keeps model copies in sync., interconnect bandwidth often determines how efficiently a cluster is used, and it has grown more slowly than compute (see memory wallmemory wall, n.: The growing gap between how fast chips can calculate and how fast they can fetch data from memory. Processors increasingly sit idle, waiting for numbers to arrive.).
- 2
Chip design
The metal wiring layers that connect transistors and functional blocks within an integrated circuit. As features shrink, wire resistance and capacitance increasingly limit speed and power, making interconnect a central concern of process nodeprocess node, n.: The label for a generation of chip-making technology, like '3 nanometer'. Smaller numbers mean newer, denser chips, but the number no longer measures any actual part of the chip. design.
Origin
A general electronics term for wiring between components, extended to the networks of multiprocessor supercomputers by the 1980s. In AI it gained prominence as clusters grew from a few GPUs to hundreds of thousands, with Nvidia's NVLink (2014) and its acquisition of Mellanox (completed 2020) marking interconnect as a competitive battleground.
Usage note
The Ultra Ethernet Consortium (2023) and UALink Consortium (2024) were formed largely to offer open alternatives to Nvidia's NVLink and InfiniBand.
Related
- NVLinkNVLink, prop. n.: Nvidia's private high-speed connection between its AI chips, much faster than ordinary computer connections, letting dozens of chips share memory and act almost like one giant chip.
- InfiniBandInfiniBand, prop. n.: A very fast networking technology used to connect the thousands of servers in supercomputers and AI training clusters, designed so data moves with minimal delay.
- RDMARDMA, abbr.: A way for one computer to read or write another computer's memory directly over the network, skipping the usual software middlemen, so data moves faster with less delay.
- scale-up and scale-outscale-up and scale-out, phr.: Two ways to get more computing power: make each machine bigger and more tightly linked (scale up), or add more machines connected by a network (scale out).
- all-reduceall-reduce, n.: A group operation in which every chip contributes a set of numbers, the sets are added together, and every chip ends up with the same total. It is how parallel training keeps model copies in sync.
- collective communicationcollective communication, n.: The standard patterns by which a whole group of chips exchanges data at once — for example, everyone adding up their numbers and all getting the total — which large training jobs use constantly.
- advanced packagingadvanced packaging, n.: Ways of joining several chips very closely inside one package — side by side on a silicon base or stacked on top of each other — so they act almost like one bigger chip.
- photonic computingphotonic computing, n.: Using light instead of electricity to compute or move data. Light can carry huge amounts of information with little energy, but building full computers from it is still very hard.
Mentioned in
AI acceleratorAI accelerator, n.: Any chip built or tuned to speed up AI — graphics chips, Google's TPUs, phone 'neural engines' and many startup designs. Such chips do the heavy math of training and running models., computecompute, n.: The raw computing power used to train and run AI, usually measured by how many specialized chips are used and for how long. With data and algorithms, it is a main driver of AI progress., AI supercomputerAI supercomputer, n.: A huge network of AI chips — often tens or hundreds of thousands — wired together in a data center to train the largest AI models., neocloudneocloud, n.: A newer kind of cloud company that mainly rents out powerful AI chips, rather than the full range of computing services that giants like Amazon or Microsoft sell., advanced packagingadvanced packaging, n.: Ways of joining several chips very closely inside one package — side by side on a silicon base or stacked on top of each other — so they act almost like one bigger chip., all-reduceall-reduce, n.: A group operation in which every chip contributes a set of numbers, the sets are added together, and every chip ends up with the same total. It is how parallel training keeps model copies in sync., chipletchiplet, n.: One of several smaller chips that are made separately and then joined inside a single package to work as one processor, which is cheaper than making one giant chip and lets makers reuse parts., collective communicationcollective communication, n.: The standard patterns by which a whole group of chips exchanges data at once — for example, everyone adding up their numbers and all getting the total — which large training jobs use constantly., decentralized trainingdecentralized training, n.: Training an AI model on computers owned by many different people or companies around the world, linked over the ordinary internet rather than housed together in one data center., disaggregated servingdisaggregated serving, n.: Running the two stages of an AI's reply, reading the prompt and writing the answer, on separate groups of chips, each tuned for its own job, and handing the work between them., Huang's lawHuang's law, n.: A claim, named after Nvidia's chief executive, that AI chip performance improves much faster than Moore's law because chips, software and number formats all improve together., InfiniBandInfiniBand, prop. n.: A very fast networking technology used to connect the thousands of servers in supercomputers and AI training clusters, designed so data moves with minimal delay., NVLinkNVLink, prop. n.: Nvidia's private high-speed connection between its AI chips, much faster than ordinary computer connections, letting dozens of chips share memory and act almost like one giant chip., photonic computingphotonic computing, n.: Using light instead of electricity to compute or move data. Light can carry huge amounts of information with little energy, but building full computers from it is still very hard., RDMARDMA, abbr.: A way for one computer to read or write another computer's memory directly over the network, skipping the usual software middlemen, so data moves faster with less delay., roofline modelroofline model, n.: A simple chart showing the fastest a program can run on a chip: limited either by how quickly data arrives from memory or by the chip's top calculating speed., scale-up and scale-outscale-up and scale-out, phr.: Two ways to get more computing power: make each machine bigger and more tightly linked (scale up), or add more machines connected by a network (scale out).