decode phase
In plain wordsThe second stage of an AI's reply, in which it writes the answer one word-piece at a time, each step depending on everything written so far.
The phase of autoregressiveautoregressive model, n.: A model that produces a sequence one piece at a time, each new piece depending on the pieces already produced — like writing a sentence word by word without going back. generation that follows prefillprefill, n.: The first stage of an AI's reply, in which it reads your whole prompt in one go before it starts writing. Long prompts make this stage take longer.: the model repeatedly runs a forward passforward pass, n.: The step where a neural network takes an input and runs it through its layers, one after another, to produce an output — its answer or prediction. on the newest tokentoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens. of each sequence, attends over the keys and values stored in the KV cacheKV cache, n.: A memory store where a chatbot keeps its working notes on everything already read and written, so each new word does not require re-reading the whole conversation from scratch., appends the new token's keys and values, and selects the next token, until a stop condition is met. Each step does little arithmetic per byte moved, because the full weightsweights, n.: The learned numbers inside a neural network that set how strongly each part influences the next. The finished set of weights is, in effect, the trained model itself. and the KV cache must be read from memory to produce one token per sequence, so decoding is typically memory-boundmemory-bound, adj.: Describes a computer task slowed by how fast data can be fetched from memory rather than by how fast the chip can calculate — like a fast cook kept waiting for deliveries. and its per-request speed is limited by memory bandwidthmemory bandwidth, n.: How fast data can be moved between a chip's memory and its calculators. For chatbots, this often decides how quickly words appear, more than raw calculating speed.. Batching many sequences, quantizationquantization, n.: Storing an AI model's numbers with fewer digits, like rounding prices to the nearest dollar, so the model takes less memory and runs faster, usually with only a small loss of accuracy., speculative decodingspeculative decoding, n.: A speed-up trick in which a small, fast model guesses the next few words and the big model checks all the guesses at once, keeping the right ones: faster answers, same quality., grouped-query attentiongrouped-query attention, n.: A middle ground between full multi-head attention and its cheapest variant: attention heads are split into groups, and each group shares one store of information about earlier words, saving memory with little loss in quality. and multi-token predictionmulti-token prediction, n.: Training a language model to guess several upcoming words at once rather than only the next one, which can help it learn better and later write faster. all aim to get more useful tokens out of each memory read. Decode speed sets time per output tokentime per output token, n.: Once an AI has started replying, the average wait between one piece of its answer and the next. It sets how quickly the text seems to flow..
At batch size one, the decode phase leaves most of the GPU's arithmetic units idle.
Origin
Grew out of 'incremental decoding' in sequence-to-sequence systems. Noam Shazeer's 2019 paper on multi-query attention identified the memory bandwidth spent repeatedly loading keys and values as its bottleneck. 'Decode' as the name of the phase paired with 'prefill' became standard in LLM-serving papers in 2023–2024.
Usage note
Not to be confused with decodingdecoding, n.: The method a language model uses to turn its predictions into actual text — for example, always picking the most likely next word, or choosing somewhat at random among the likely ones., the strategy for choosing tokens (greedy decodinggreedy decoding, n.: The simplest way for a language model to write: always pick the single most likely next word. It is fast and predictable but can produce dull or repetitive text., beam searchbeam search, n.: A search method that keeps several of the best partial answers at each step, rather than only the single best, to find a better overall result without trying every possibility., sampling), nor with the decoder half of an encoder–decoder architectureencoder–decoder architecture, n.: A two-part model design: one part, the encoder, reads the input and condenses it into an internal summary; the other, the decoder, uses that summary to produce the output..
Related
- prefillprefill, n.: The first stage of an AI's reply, in which it reads your whole prompt in one go before it starts writing. Long prompts make this stage take longer.
- KV cacheKV cache, n.: A memory store where a chatbot keeps its working notes on everything already read and written, so each new word does not require re-reading the whole conversation from scratch.
- memory-boundmemory-bound, adj.: Describes a computer task slowed by how fast data can be fetched from memory rather than by how fast the chip can calculate — like a fast cook kept waiting for deliveries.
- time per output tokentime per output token, n.: Once an AI has started replying, the average wait between one piece of its answer and the next. It sets how quickly the text seems to flow.
- speculative decodingspeculative decoding, n.: A speed-up trick in which a small, fast model guesses the next few words and the big model checks all the guesses at once, keeping the right ones: faster answers, same quality.
- continuous batchingcontinuous batching, n.: A way for an AI server to handle many conversations together, letting new ones join and finished ones leave after every word, instead of waiting for a whole group to finish.
- prefillprefill, n.: The first stage of an AI's reply, in which it reads your whole prompt in one go before it starts writing. Long prompts make this stage take longer.
- decodingdecoding, n.: The method a language model uses to turn its predictions into actual text — for example, always picking the most likely next word, or choosing somewhat at random among the likely ones.
Sources
- Shazeer, N. (2019). Fast Transformer Decoding: One Write-Head is All You Need. arXiv:1911.02150. arxiv.org/abs/1911.02150
Mentioned in
inference costinference cost, n.: What it costs to use an AI model after it has been built: the computing power, electricity and hardware needed every time it answers, usually billed per chunk of text., decodingdecoding, n.: The method a language model uses to turn its predictions into actual text — for example, always picking the most likely next word, or choosing somewhat at random among the likely ones., forward passforward pass, n.: The step where a neural network takes an input and runs it through its layers, one after another, to produce an output — its answer or prediction., inferenceinference, n.: Drawing a conclusion from evidence. In today's AI it most often means running an already-trained model to get an answer, as opposed to training it., KV cacheKV cache, n.: A memory store where a chatbot keeps its working notes on everything already read and written, so each new word does not require re-reading the whole conversation from scratch., latencylatency, n.: How long you wait between asking an AI system for something and getting the answer. Lower latency means a snappier, more responsive system., memory bandwidthmemory bandwidth, n.: How fast data can be moved between a chip's memory and its calculators. For chatbots, this often decides how quickly words appear, more than raw calculating speed., prefillprefill, n.: The first stage of an AI's reply, in which it reads your whole prompt in one go before it starts writing. Long prompts make this stage take longer., speculative decodingspeculative decoding, n.: A speed-up trick in which a small, fast model guesses the next few words and the big model checks all the guesses at once, keeping the right ones: faster answers, same quality., tokens per secondtokens per second, n.: A speed rating for AI text generation: how many word-pieces a model produces each second. Five to ten keeps up with a fast reader; hundreds feel instant., arithmetic intensityarithmetic intensity, n.: How much calculating a task does for each piece of data it fetches from memory. Tasks with little calculation per fetch are slowed by memory, not by the chip's math speed., chunked prefillchunked prefill, n.: A serving trick that splits the reading of a long prompt into smaller pieces and slots them between other users' replies, so nobody's answer freezes while a huge document is read., disaggregated servingdisaggregated serving, n.: Running the two stages of an AI's reply, reading the prompt and writing the answer, on separate groups of chips, each tuned for its own job, and handing the work between them., memory-boundmemory-bound, adj.: Describes a computer task slowed by how fast data can be fetched from memory rather than by how fast the chip can calculate — like a fast cook kept waiting for deliveries., roofline modelroofline model, n.: A simple chart showing the fastest a program can run on a chip: limited either by how quickly data arrives from memory or by the chip's top calculating speed., time per output tokentime per output token, n.: Once an AI has started replying, the average wait between one piece of its answer and the next. It sets how quickly the text seems to flow.