Skip to content
Last Dictionary
Plane
AI Engineering & MLOpsInference & EfficiencyLanguage Models & Transformers

semantic caching

nounAlso: semantic cache, LLM response caching, GPTCache

In plain wordsSaving an AI system's past answers and reusing one whenever a new question means the same as an old one, even if worded differently, to save time and money.

  1. 1

    Language-model applications

    A caching technique that returns a stored model response when a new request is similar in meaning, not merely identical in text, to an earlier one. The application computes an embeddingembedding, n.: A list of numbers that stands for a word, picture or other item, arranged so that similar items get similar lists. It lets a computer measure similarity of meaning as distance. of each incoming prompt and searches a vector indexvector database, n.: A database built to store embeddings — lists of numbers that represent meaning — and to quickly find the stored items most similar to a new one. It is the memory behind many AI search tools. of previous prompts; if the nearest neighbor exceeds a similarity threshold, the cached answer is returned without calling the model, and otherwise the model is called and the new pair stored. Hits cut latencylatency, n.: How long you wait between asking an AI system for something and getting the answer. Lower latency means a snappier, more responsive system. from seconds to milliseconds and avoid per-tokentoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens. charges, which pays off for repetitive traffic such as customer-support questions. The central risk is a false hit: 'cancel my order' and 'don't cancel my order' can embed close together, and answers that depend on the user, the date or the conversation must not be shared. Systems therefore set conservative thresholds, scope caches per user or context, and expire entries.

    Semantic caching answered 30 percent of support questions, but we excluded anything mentioning an account number.

  2. 2

    Databases (earlier sense)

    A client-side caching method for database and client–server systems in which cached query results are indexed by the queries, or semantic regions, that produced them rather than by pages or tuples, so that a new query can be answered partly from the cache and partly by a smaller remainder query sent to the server.

Origin

The database sense was introduced by Shaul Dar, Michael Franklin and colleagues at the VLDB conference in 1996. The language-model sense arose in 2023, when open-source tools such as Zilliz's GPTCache and vector-database vendors offered embedding-based response caches; AI gatewaysAI gateway, n.: A checkpoint between an organization's apps and the AI models they use that handles access, costs, logging, safety checks and switching between providers in one place. later built them in.

Usage note

Different from prompt cachingprompt caching, n.: Saving the work a model did reading the start of a prompt, so later requests that begin the same way can skip that reading and come back faster and cheaper., in which a provider reuses the computed key–value cacheKV cache, n.: A memory store where a chatbot keeps its working notes on everything already read and written, so each new word does not require re-reading the whole conversation from scratch. for an identical prompt prefix but still generates a fresh answer; semantic caching skips generation entirely.

Related

See also
  • prompt cachingprompt caching, n.: Saving the work a model did reading the start of a prompt, so later requests that begin the same way can skip that reading and come back faster and cheaper.
  • embeddingembedding, n.: A list of numbers that stands for a word, picture or other item, arranged so that similar items get similar lists. It lets a computer measure similarity of meaning as distance.
  • vector databasevector database, n.: A database built to store embeddings — lists of numbers that represent meaning — and to quickly find the stored items most similar to a new one. It is the memory behind many AI search tools.
  • AI gatewayAI gateway, n.: A checkpoint between an organization's apps and the AI models they use that handles access, costs, logging, safety checks and switching between providers in one place.
  • KV cacheKV cache, n.: A memory store where a chatbot keeps its working notes on everything already read and written, so each new word does not require re-reading the whole conversation from scratch.
  • inference costinference cost, n.: What it costs to use an AI model after it has been built: the computing power, electricity and hardware needed every time it answers, usually billed per chunk of text.
Contrast with
  • prompt cachingprompt caching, n.: Saving the work a model did reading the start of a prompt, so later requests that begin the same way can skip that reading and come back faster and cheaper.

Sources

  1. Dar, S., Franklin, M. J., Jónsson, B. T., Srivastava, D. & Tan, M. (1996). Semantic Data Caching and Replacement. Proceedings of the 22nd VLDB Conference.

Mentioned in

prompt cachingprompt caching, n.: Saving the work a model did reading the start of a prompt, so later requests that begin the same way can skip that reading and come back faster and cheaper., text embeddingtext embedding, n.: A list of numbers that captures what a piece of text means, so that texts with similar meanings get similar lists. Computers use them to search, sort and group text by meaning., AI gatewayAI gateway, n.: A checkpoint between an organization's apps and the AI models they use that handles access, costs, logging, safety checks and switching between providers in one place.