Inference & EfficiencyLanguage Models & TransformersHardware & Compute

KV cache

nounAlso: key-value cache, attention cache, past key values

In plain wordsA memory store where a chatbot keeps its working notes on everything already read and written, so each new word does not require re-reading the whole conversation from scratch.

The stored key and valuequery, key and value, n.: The three roles each word plays inside attention: the query is what it is looking for, the key is what it can be found by, and the value is the information it hands over when chosen. vectors computed for earlier tokenstoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens. in every attentionattention, n.: A way for an AI model to choose what to focus on. For each word it processes, the model scores how relevant every other word is and draws information mostly from the relevant ones. layer of a transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it., kept during generation so that each decodedecode phase, n.: The second stage of an AI's reply, in which it writes the answer one word-piece at a time, each step depending on everything written so far. step computes keys and values only for the newest token instead of recomputing them for the whole sequence. This makes each step's cost linear rather than quadratic in sequence length, at the price of memory that grows with sequence length, batch size, layer count and number of key–value heads. In the worked example of the vLLMvLLM, prop. n.: A free, open-source program for running large language models on servers efficiently, so one machine can answer many more users at once. Many companies use it to host AI models. paper, OPT-13B in 16-bit precision needs about 800 KB per token, or about 1.6 GB for a 2,048-token request. In serving, the KV cache often rivals or exceeds the weightsweights, n.: The learned numbers inside a neural network that set how strongly each part influences the next. The finished set of weights is, in effect, the trained model itself. in size and limits batch size and long contextlong context, n.: The ability of an AI model to work with very long inputs — whole books, codebases or hours of transcripts — in one go, and the techniques that make this possible.. It is managed with PagedAttentionPagedAttention, prop. n.: A way of storing an AI model's working memory in small fixed-size blocks, like the pages of a computer's memory, so little space is wasted and more users can be served at once., reused across requests by prompt cachingprompt caching, n.: Saving the work a model did reading the start of a prompt, so later requests that begin the same way can skip that reading and come back faster and cheaper., shrunk by grouped-query attentiongrouped-query attention, n.: A middle ground between full multi-head attention and its cheapest variant: attention heads are split into groups, and each group shares one store of information about earlier words, saving memory with little loss in quality., multi-head latent attentionmulti-head latent attention, n.: An attention design from the Chinese lab DeepSeek that compresses the stored information about earlier words into a much smaller summary, so long conversations need far less memory. and KV cache compressionKV cache compression, n.: Techniques that shrink the memory an AI model uses to remember a long conversation or document, by storing its notes more compactly or discarding the least useful ones., and transferred between machines in disaggregated servingdisaggregated serving, n.: Running the two stages of an AI's reply, reading the prompt and writing the answer, on separate groups of chips, each tuned for its own job, and handing the work between them..
Switching to grouped-query attention cut our KV cache by a factor of eight.

\text{bytes} = 2 \cdot n_{\text{layers}} \cdot n_{\text{kv heads}} \cdot d_{\text{head}} \cdot n_{\text{tokens}} \cdot n_{\text{bytes per value}}

Origin

Caching keys and values during incremental decoding was standard in Transformer implementations from about 2017–2018, often under names like 'past key values'. The short name 'KV cache', used in Google's 2022 analysis of PaLM inference by Reiner Pope and colleagues, became common around 2022–2023, when LLM-serving work such as vLLM showed that its memory footprint, not arithmetic, limited how many requests a GPU could serve.

Sources

Kwon, W. et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP 2023. arXiv:2309.06180. arxiv.org/abs/2309.06180
Shazeer, N. (2019). Fast Transformer Decoding: One Write-Head is All You Need. arXiv:1911.02150. arxiv.org/abs/1911.02150
Pope, R. et al. (2022). Efficiently Scaling Transformer Inference. arXiv:2211.05102 (MLSys 2023). arxiv.org/abs/2211.05102

Mentioned in

context windowcontext window, n.: How much an AI model can read and keep in view at once: the instructions, the conversation so far and any attached documents. Text that does not fit has to be cut, summarized or left out., optimizationoptimization, n.: Finding the best possible choice according to some score, such as the lowest error or the highest reward. Training an AI is optimization: adjusting its settings until its mistakes are as small as possible., big O notationbig O notation, n.: Shorthand for how the cost of a computation grows as the problem gets bigger. Saying attention costs 'O(n²)' means doubling the length of the text roughly quadruples the work., causal maskcausal mask, n.: A rule inside text-generating AI that stops each word from seeing the words after it, so the model learns to predict what comes next without being able to cheat., decode phasedecode phase, n.: The second stage of an AI's reply, in which it writes the answer one word-piece at a time, each step depending on everything written so far., decoder-only transformerdecoder-only transformer, n.: The kind of transformer behind most chatbots: a single stack that reads text left to right and continues it, without a separate part that first encodes an input., forward passforward pass, n.: The step where a neural network takes an input and runs it through its layers, one after another, to produce an output — its answer or prediction., high-bandwidth memoryhigh-bandwidth memory, n.: Very fast memory made by stacking memory chips on top of each other and placing them right next to the processor, so AI chips can be fed data quickly enough., inferenceinference, n.: Drawing a conclusion from evidence. In today's AI it most often means running an already-trained model to get an answer, as opposed to training it., inference engineinference engine, n.: The part of an AI system that does the working-out: in older AI, the program that applies rules to facts; in today's AI, the software that runs a trained model to produce answers., long contextlong context, n.: The ability of an AI model to work with very long inputs — whole books, codebases or hours of transcripts — in one go, and the techniques that make this possible., memory bandwidthmemory bandwidth, n.: How fast data can be moved between a chip's memory and its calculators. For chatbots, this often decides how quickly words appear, more than raw calculating speed., model servingmodel serving, n.: Running a trained AI model as a live service, so that apps and people can send it requests and get answers back quickly, reliably and at a sensible cost., multi-head attentionmulti-head attention, n.: Running several attention processes side by side, each free to track a different kind of relationship between words — grammar, topic, who did what — and then combining their results., prefillprefill, n.: The first stage of an AI's reply, in which it reads your whole prompt in one go before it starts writing. Long prompts make this stage take longer., prompt cachingprompt caching, n.: Saving the work a model did reading the start of a prompt, so later requests that begin the same way can skip that reading and come back faster and cheaper., quantizationquantization, n.: Storing an AI model's numbers with fewer digits, like rounding prices to the nearest dollar, so the model takes less memory and runs faster, usually with only a small loss of accuracy., query, key and valuequery, key and value, n.: The three roles each word plays inside attention: the query is what it is looking for, the key is what it can be found by, and the value is the information it hands over when chosen., throughputthroughput, n.: How much work an AI system gets done per second across all its users at once — for a chatbot service, roughly how much text it can produce in total each second., vLLMvLLM, prop. n.: A free, open-source program for running large language models on servers efficiently, so one machine can answer many more users at once. Many companies use it to host AI models., VRAMVRAM, abbr.: The memory on a graphics card. For running AI at home it is the key number: the model has to fit in it, so more memory means bigger models., attention sinkattention sink, n.: A quirk of language models: they pour a large share of their attention onto the very first word of the text, even when it means nothing, as a place to park attention they do not need., cache-augmented generationcache-augmented generation, n.: An alternative to search-based AI answering: load the whole knowledge base, if it is small enough, into the model's context once, save the model's processed state, and reuse it for every question., diffusion language modeldiffusion language model, n.: A language model that writes by starting from a fully blanked-out answer and filling it in over several rounds, settling many words at once instead of writing strictly left to right.

KV cache

Origin

Related

Sources

Mentioned in