scale-up and scale-out
In plain wordsTwo ways to get more computing power: make each machine bigger and more tightly linked (scale up), or add more machines connected by a network (scale out).
Complementary ways of growing a computing system. In general computing, scaling up (vertical scaling) means a bigger single machine, and scaling out (horizontal scaling) means more machines working in parallel. In AI clusters the terms name two network tiers. The scale-up domain is a group of accelerators — 8 in a classic GPU server, 72 in a GB200 NVL72 rack, thousands in a TPU pod — joined by very-high-bandwidth, memory-semantic links such as NVLinkNVLink, prop. n.: Nvidia's private high-speed connection between its AI chips, much faster than ordinary computer connections, letting dozens of chips share memory and act almost like one giant chip. that let them behave almost as one device, which suits tensor parallelismtensor parallelism, n.: Splitting the individual giant tables of numbers inside each layer of a model across several chips, so that every chip does a slice of every calculation at the same time. and expert parallelismexpert parallelism, n.: A way of running a 'mixture of experts' model — one made of many specialist sub-networks — by giving each chip its own set of specialists and sending each piece of text to whichever chip holds the specialists it needs.. The scale-out network (InfiniBandInfiniBand, prop. n.: A very fast networking technology used to connect the thousands of servers in supercomputers and AI training clusters, designed so data moves with minimal delay. or Ethernet with RDMARDMA, abbr.: A way for one computer to read or write another computer's memory directly over the network, skipping the usual software middlemen, so data moves faster with less delay.) connects many such domains with less bandwidth per accelerator and carries data-paralleldata parallelism, n.: Splitting the training data among many chips that each hold a full copy of the model; every chip learns from its own share, and they pool what they learned after each step so all copies stay identical. and pipeline-parallelpipeline parallelism, n.: Splitting a model's layers into consecutive stages on different chips, like stations on an assembly line, and streaming small batches of data through so that every station stays busy. traffic. Since 2025, 'scale-across' has named the linking of several data centers into one training system.
Keep tensor parallelism inside the scale-up domain; anything that crosses the scale-out network kills throughput.
Origin
The scale-up/scale-out distinction comes from server and database architecture of the 1990s and 2000s. Its use for the network tiers of AI clusters spread around 2023–2024 as rack-scale systemsrack-scale system, n.: An AI computer designed as one whole cabinet of chips, cooling and wiring, rather than separate servers, so all the chips inside can work together like one big chip. enlarged scale-up domains; Nvidia introduced 'scale-across' with its Spectrum-XGS Ethernet in August 2025.
Usage note
The boundary is moving: optical links, UALink and larger switches aim to stretch scale-up domains to hundreds or thousands of accelerators.
Related
- interconnectinterconnect, n.: The wiring and networking that connect chips to each other — inside a server, across a data center, or within a single chip — so they can share data and work as one.
- NVLinkNVLink, prop. n.: Nvidia's private high-speed connection between its AI chips, much faster than ordinary computer connections, letting dozens of chips share memory and act almost like one giant chip.
- InfiniBandInfiniBand, prop. n.: A very fast networking technology used to connect the thousands of servers in supercomputers and AI training clusters, designed so data moves with minimal delay.
- rack-scale systemrack-scale system, n.: An AI computer designed as one whole cabinet of chips, cooling and wiring, rather than separate servers, so all the chips inside can work together like one big chip.
- distributed trainingdistributed training, n.: Training one AI model on many computer chips at once, splitting up the data or the model itself among them and keeping them in sync, so that jobs that would take a single chip thousands of years finish in weeks.
- AI supercomputerAI supercomputer, n.: A huge network of AI chips — often tens or hundreds of thousands — wired together in a data center to train the largest AI models.
Mentioned in
interconnectinterconnect, n.: The wiring and networking that connect chips to each other — inside a server, across a data center, or within a single chip — so they can share data and work as one., InfiniBandInfiniBand, prop. n.: A very fast networking technology used to connect the thousands of servers in supercomputers and AI training clusters, designed so data moves with minimal delay., NVLinkNVLink, prop. n.: Nvidia's private high-speed connection between its AI chips, much faster than ordinary computer connections, letting dozens of chips share memory and act almost like one giant chip., rack-scale systemrack-scale system, n.: An AI computer designed as one whole cabinet of chips, cooling and wiring, rather than separate servers, so all the chips inside can work together like one big chip., RDMARDMA, abbr.: A way for one computer to read or write another computer's memory directly over the network, skipping the usual software middlemen, so data moves faster with less delay.