cache-augmented generation

An alternative to retrieval-augmented generationretrieval-augmented generation, n.: Giving an AI an open-book exam: before answering, it looks up relevant passages in a set of documents and bases its reply on them, often citing where each point came from. for small knowledge bases: instead of searching for passages at every query, the whole collection is loaded once into a long-contextlong context, n.: The ability of an AI model to work with very long inputs — whole books, codebases or hours of transcripts — in one go, and the techniques that make this possible. model's context windowcontext window, n.: How much an AI model can read and keep in view at once: the instructions, the conversation so far and any attached documents. Text that does not fit has to be cut, summarized or left out., the model's intermediate state for that text (its KV cacheKV cache, n.: A memory store where a chatbot keeps its working notes on everything already read and written, so each new word does not require re-reading the whole conversation from scratch.) is saved, and each question is answered by appending it to the saved state. With no search step there is nothing to chunkchunking, n.: Cutting long documents into smaller pieces before storing them for AI search, so the system can find and hand the model just the relevant part instead of a whole manual. or index and no retrieval to wait for or get wrong, and prompt cachingprompt caching, n.: Saving the work a model did reading the start of a prompt, so later requests that begin the same way can skip that reading and come back faster and cheaper. makes the reuse cheap. Its limits are those of long contexts — the knowledge base must fit, costs grow with its size, and context rotcontext rot, n.: The way an AI model becomes less reliable as you give it more and more text to read, even when the answer is in there somewhere. and lost in the middlelost in the middle, phr.: AI models reading a long input behave a little like a skimmer: they reliably catch facts placed near its start or end but are much likelier to miss the same fact buried halfway through. effects still apply — so it suits small, stable collections such as a product manual or a set of policies, while retrieval-augmented generation remains preferable for large or fast-changing corpora.

Our handbook is only 80,000 tokens, so cache-augmented generation beat the RAG pipeline on both latency and accuracy.