prompt caching
In plain wordsSaving the work a model did reading the start of a prompt, so later requests that begin the same way can skip that reading and come back faster and cheaper.
Reuse of the KV cacheKV cache, n.: A memory store where a chatbot keeps its working notes on everything already read and written, so each new word does not require re-reading the whole conversation from scratch. computed during prefillprefill, n.: The first stage of an AI's reply, in which it reads your whole prompt in one go before it starts writing. Long prompts make this stage take longer. for a prompt prefix, such as a system promptsystem prompt, n.: Background instructions given to a chatbot before the conversation starts, setting its role, rules, tone and tools. Users usually never see it, but it shapes every reply., tool definitions, a long document or earlier conversation turns, by later requests that begin with exactly the same tokens, so that only the new suffix needs processing. Inside serving engines it is called prefix caching, implemented by hashing fixed-size blocks of tokens (automatic prefix caching in vLLMvLLM, prop. n.: A free, open-source program for running large language models on servers efficiently, so one machine can answer many more users at once. Many companies use it to host AI models.) or indexing cached sequences in a radix tree (RadixAttention in SGLangSGLang, prop. n.: A free, open-source engine for running AI language and image models on servers, built to be fast when many requests begin the same way, as in chat and agent workloads.), with least-recently-used eviction. Model APIsmodel API, n.: A web service through which programs send requests to an AI model run by a company and receive its answers, usually paying per use instead of running the model themselves. expose it as a priced feature: cached input tokens are billed at a fraction of the normal rate, either automatically or at breakpoints the developer marks, and entries expire after a period of disuse, typically minutes to an hour. It sharply cuts time to first tokentime to first token, n.: How long a chatbot takes to start answering: the pause between pressing enter and seeing the first word appear. and cost for agents, chat and retrieval workloads, but only exact prefix matches count, so prompts are arranged with stable content first and changing content last.
We moved the timestamp to the end of the system prompt and our cache hit rate jumped to 95 percent.
Origin
Prefix sharing appeared in serving research in 2023, in vLLM's shared memory blocks and SGLang's RadixAttention (December 2023). Commercial APIs followed in 2024: Google offered context caching for Gemini in mid-2024, Anthropic launched prompt caching in August 2024 (cache reads at 10% of the base input price, writes at a 25% premium), and OpenAI added automatic caching in October 2024.
Usage note
Not the same as semantic cachingsemantic caching, n.: Saving an AI system's past answers and reusing one whenever a new question means the same as an old one, even if worded differently, to save time and money., which returns a stored answer to a similar question without running the model; prompt caching still runs the model, so outputs are unaffected. Caches shared across users can leak information through response timing, so providers typically isolate them per organization.
Related
- KV cacheKV cache, n.: A memory store where a chatbot keeps its working notes on everything already read and written, so each new word does not require re-reading the whole conversation from scratch.
- prefillprefill, n.: The first stage of an AI's reply, in which it reads your whole prompt in one go before it starts writing. Long prompts make this stage take longer.
- time to first tokentime to first token, n.: How long a chatbot takes to start answering: the pause between pressing enter and seeing the first word appear.
- inference costinference cost, n.: What it costs to use an AI model after it has been built: the computing power, electricity and hardware needed every time it answers, usually billed per chunk of text.
- SGLangSGLang, prop. n.: A free, open-source engine for running AI language and image models on servers, built to be fast when many requests begin the same way, as in chat and agent workloads.
- context engineeringcontext engineering, n.: Managing an AI model's reading material: choosing, trimming and arranging the instructions, documents, tool results and memories it is given on every turn, so that it has what the task needs and little else.
- semantic cachingsemantic caching, n.: Saving an AI system's past answers and reusing one whenever a new question means the same as an old one, even if worded differently, to save time and money.
Sources
- Anthropic (2024). Prompt caching with Claude. Anthropic announcement, 14 August 2024. claude.com/blog/prompt-caching
- Zheng, L. et al. (2023). SGLang: Efficient Execution of Structured Language Model Programs. NeurIPS 2024. arXiv:2312.07104. arxiv.org/abs/2312.07104
Mentioned in
inference costinference cost, n.: What it costs to use an AI model after it has been built: the computing power, electricity and hardware needed every time it answers, usually billed per chunk of text., model APImodel API, n.: A web service through which programs send requests to an AI model run by a company and receive its answers, usually paying per use instead of running the model themselves., compactioncompaction, n.: Like replacing a stack of rough notes with a one-page digest: when an AI's conversation grows too long for its memory, the history is condensed into a summary and the work continues from there., KV cacheKV cache, n.: A memory store where a chatbot keeps its working notes on everything already read and written, so each new word does not require re-reading the whole conversation from scratch., prefillprefill, n.: The first stage of an AI's reply, in which it reads your whole prompt in one go before it starts writing. Long prompts make this stage take longer., time to first tokentime to first token, n.: How long a chatbot takes to start answering: the pause between pressing enter and seeing the first word appear., vLLMvLLM, prop. n.: A free, open-source program for running large language models on servers efficiently, so one machine can answer many more users at once. Many companies use it to host AI models., AI gatewayAI gateway, n.: A checkpoint between an organization's apps and the AI models they use that handles access, costs, logging, safety checks and switching between providers in one place., cache-augmented generationcache-augmented generation, n.: An alternative to search-based AI answering: load the whole knowledge base, if it is small enough, into the model's context once, save the model's processed state, and reuse it for every question., contextual retrievalcontextual retrieval, n.: A fix for searching cut-up documents: before storing each piece, an AI writes a short note saying where it came from and what it is about, so the piece still makes sense on its own., PagedAttentionPagedAttention, prop. n.: A way of storing an AI model's working memory in small fixed-size blocks, like the pages of a computer's memory, so little space is wasted and more users can be served at once., semantic cachingsemantic caching, n.: Saving an AI system's past answers and reusing one whenever a new question means the same as an old one, even if worded differently, to save time and money., SGLangSGLang, prop. n.: A free, open-source engine for running AI language and image models on servers, built to be fast when many requests begin the same way, as in chat and agent workloads., sleep-time computesleep-time compute, n.: Letting an AI think about your documents or project while it would otherwise sit idle, before you ask anything, so it can answer later questions faster and more cheaply.