disaggregated serving
In plain wordsRunning the two stages of an AI's reply, reading the prompt and writing the answer, on separate groups of chips, each tuned for its own job, and handing the work between them.
An LLM servingmodel serving, n.: Running a trained AI model as a live service, so that apps and people can send it requests and get answers back quickly, reliably and at a sensible cost. architecture that gives the two phases of each request their own pools of accelerators: prefillprefill, n.: The first stage of an AI's reply, in which it reads your whole prompt in one go before it starts writing. Long prompts make this stage take longer. runs on one set, and the request's KV cacheKV cache, n.: A memory store where a chatbot keeps its working notes on everything already read and written, so each new word does not require re-reading the whole conversation from scratch. is then shipped over a fast interconnectinterconnect, n.: The wiring and networking that connect chips to each other — inside a server, across a data center, or within a single chip — so they can share data and work as one. such as NVLinkNVLink, prop. n.: Nvidia's private high-speed connection between its AI chips, much faster than ordinary computer connections, letting dozens of chips share memory and act almost like one giant chip. or RDMARDMA, abbr.: A way for one computer to read or write another computer's memory directly over the network, skipping the usual software middlemen, so data moves faster with less delay. networking to another set that performs the decodedecode phase, n.: The second stage of an AI's reply, in which it writes the answer one word-piece at a time, each step depending on everything written so far.. The case for it, made by systems such as Splitwise and DistServe, is that the phases want different things — prefill is compute-bound and judged by time to first tokentime to first token, n.: How long a chatbot takes to start answering: the pause between pressing enter and seeing the first word appear., decode is memory-boundmemory-bound, adj.: Describes a computer task slowed by how fast data can be fetched from memory rather than by how fast the chip can calculate — like a fast cook kept waiting for deliveries. and judged by time per output tokentime per output token, n.: Once an AI has started replying, the average wait between one piece of its answer and the next. It sets how quickly the text seems to flow. — so sharing batches lets long prompts stall token streams and forces one parallelism setting on both. Separate pools can each be sized, parallelized and even built from different hardware for their own target. Its costs are KV-cache transfer bandwidth and more complex scheduling and capacity planning. Large deployments add tiered KV-cache stores in CPU memory and SSD, and some split further, separating attention from feed-forward or expertmixture of experts, n.: A way to build very large AI models cheaply: the model contains many specialist sub-networks, and for each word only a few are switched on, picked by a small dispatcher network. layers.
Moving to disaggregated serving let us scale decode nodes on their own when answers got longer.
Origin
Proposed independently around the turn of 2024 by Splitwise (Pratyush Patel and colleagues at the University of Washington and Microsoft, November 2023), DistServe (Yinmin Zhong and colleagues, January 2024) and others, and run at scale in Moonshot AI's Mooncake platform for its Kimi service (2024). In 2025 it became a standard feature of open-source serving stacks such as NVIDIA Dynamo and llm-d.
Usage note
The main alternative for limiting prefill–decode interference on shared hardware is chunked prefillchunked prefill, n.: A serving trick that splits the reading of a long prompt into smaller pieces and slots them between other users' replies, so nobody's answer freezes while a huge document is read.; many systems combine the two.
Related
- prefillprefill, n.: The first stage of an AI's reply, in which it reads your whole prompt in one go before it starts writing. Long prompts make this stage take longer.
- decode phasedecode phase, n.: The second stage of an AI's reply, in which it writes the answer one word-piece at a time, each step depending on everything written so far.
- KV cacheKV cache, n.: A memory store where a chatbot keeps its working notes on everything already read and written, so each new word does not require re-reading the whole conversation from scratch.
- chunked prefillchunked prefill, n.: A serving trick that splits the reading of a long prompt into smaller pieces and slots them between other users' replies, so nobody's answer freezes while a huge document is read.
- goodputgoodput, n.: The share of a big training job's time and computing that actually moves training forward, after subtracting time lost to crashes, restarts, waiting and repeated work. When serving users, it means the rate of requests answered fast enough.
- model servingmodel serving, n.: Running a trained AI model as a live service, so that apps and people can send it requests and get answers back quickly, reliably and at a sensible cost.
- offloadingoffloading, n.: Moving part of an AI model's data or work out of a chip's scarce fast memory onto ordinary memory, a disk or another computer, so that models too big for the chip can still run.
- chunked prefillchunked prefill, n.: A serving trick that splits the reading of a long prompt into smaller pieces and slots them between other users' replies, so nobody's answer freezes while a huge document is read.
Sources
- Patel, P., Choukse, E., Zhang, C., Shah, A., Goiri, Í., Maleki, S. & Bianchini, R. (2023). Splitwise: Efficient generative LLM inference using phase splitting. arXiv:2311.18677 (ISCA 2024). arxiv.org/abs/2311.18677
- Zhong, Y., Liu, S., Chen, J., Hu, J., Zhu, Y., Liu, X., Jin, X. & Zhang, H. (2024). DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. OSDI 2024. arXiv:2401.09670. arxiv.org/abs/2401.09670
- Qin, R. et al. (2024). Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving. arXiv:2407.00079. arxiv.org/abs/2407.00079
Mentioned in
KV cacheKV cache, n.: A memory store where a chatbot keeps its working notes on everything already read and written, so each new word does not require re-reading the whole conversation from scratch., model servingmodel serving, n.: Running a trained AI model as a live service, so that apps and people can send it requests and get answers back quickly, reliably and at a sensible cost., prefillprefill, n.: The first stage of an AI's reply, in which it reads your whole prompt in one go before it starts writing. Long prompts make this stage take longer., time to first tokentime to first token, n.: How long a chatbot takes to start answering: the pause between pressing enter and seeing the first word appear., chunked prefillchunked prefill, n.: A serving trick that splits the reading of a long prompt into smaller pieces and slots them between other users' replies, so nobody's answer freezes while a huge document is read., offloadingoffloading, n.: Moving part of an AI model's data or work out of a chip's scarce fast memory onto ordinary memory, a disk or another computer, so that models too big for the chip can still run.