AI gateway
In plain wordsA checkpoint between an organization's apps and the AI models they use that handles access, costs, logging, safety checks and switching between providers in one place.
A proxy service placed between applications and one or more model APIsmodel API, n.: A web service through which programs send requests to an AI model run by a company and receive its answers, usually paying per use instead of running the model themselves. or self-hosted models that centralizes the cross-cutting concerns of AI traffic. Typical functions are a single, often OpenAI-compatible APIOpenAI-compatible API, n.: An AI service that accepts the same request format as OpenAI's, so software written for OpenAI's models can use other companies' or self-hosted models by changing an address and key. across providers; credential management, so that applications never hold provider keys; per-team budgets, quotas and rate limitingrate limit, n.: A cap on how much a customer may use an online AI service in a given time — so many requests or words per minute — beyond which extra requests are refused until later.; routingmodel routing, n.: Automatically choosing which AI model should answer each request, a cheap quick one for easy questions and a bigger slower one for hard ones, to save money without hurting quality., load balancing, retries and failover between models or regions; cachingprompt caching, n.: Saving the work a model did reading the start of a prompt, so later requests that begin the same way can skip that reading and come back faster and cheaper., including semantic cachingsemantic caching, n.: Saving an AI system's past answers and reusing one whenever a new question means the same as an old one, even if worded differently, to save time and money.; logging and cost attribution for LLM observabilityLLM observability, n.: Recording and inspecting what an AI application actually did on each request — prompts sent, documents found, tools used, answers given, time and cost — so problems can be found and fixed.; and policy enforcement such as redaction of personal data, content filtering and guardrailsguardrails, n.: Safety checks built around an AI system — filters on what goes in and comes out, and limits on what it may do — to keep it from producing harmful or off-topic content or leaking data. on inputs and outputs. The pattern extends the API gateways of web services with token-aware metering and support for streamed responses. Open-source and commercial examples include LiteLLM, Portkey, Kong AI Gateway, Cloudflare AI Gateway and gateways from cloud providers; some also proxy Model Context ProtocolModel Context Protocol, prop. n.: An open standard that lets AI assistants plug into outside apps and data — files, calendars, databases, business software — through one common connector instead of a custom link for every pairing; often likened to a USB-C port for AI. servers and agent traffic.
All model calls go through the AI gateway, so finance sees spend by team and security can block unapproved providers.
Origin
Emerged in 2023 as companies using several language-model providers sought central control over keys, costs and logging; Cloudflare announced a product called AI Gateway that September, and API-management vendors and open-source proxies adopted the label during 2024.
Usage note
Overlaps with model routingmodel routing, n.: Automatically choosing which AI model should answer each request, a cheap quick one for easy questions and a bigger slower one for hard ones, to save money without hurting quality., which chooses a model per request; a gateway may route, but it is defined by being the shared control point. It also adds a potential single point of failure and some latencylatency, n.: How long you wait between asking an AI system for something and getting the answer. Lower latency means a snappier, more responsive system..
Related
- model APImodel API, n.: A web service through which programs send requests to an AI model run by a company and receive its answers, usually paying per use instead of running the model themselves.
- OpenAI-compatible APIOpenAI-compatible API, n.: An AI service that accepts the same request format as OpenAI's, so software written for OpenAI's models can use other companies' or self-hosted models by changing an address and key.
- model routingmodel routing, n.: Automatically choosing which AI model should answer each request, a cheap quick one for easy questions and a bigger slower one for hard ones, to save money without hurting quality.
- rate limitrate limit, n.: A cap on how much a customer may use an online AI service in a given time — so many requests or words per minute — beyond which extra requests are refused until later.
- semantic cachingsemantic caching, n.: Saving an AI system's past answers and reusing one whenever a new question means the same as an old one, even if worded differently, to save time and money.
- LLM observabilityLLM observability, n.: Recording and inspecting what an AI application actually did on each request — prompts sent, documents found, tools used, answers given, time and cost — so problems can be found and fixed.
- guardrailsguardrails, n.: Safety checks built around an AI system — filters on what goes in and comes out, and limits on what it may do — to keep it from producing harmful or off-topic content or leaking data.
Mentioned in
model APImodel API, n.: A web service through which programs send requests to an AI model run by a company and receive its answers, usually paying per use instead of running the model themselves., AI stackAI stack, n.: The layers of technology an AI product depends on, from chips and data centers at the bottom, through the models, up to the apps people use, each built on the layer below., inference providerinference provider, n.: A company that runs AI models on its own computers and sells access to them over the internet, charging by use, so customers need not buy or operate the hardware themselves., model routingmodel routing, n.: Automatically choosing which AI model should answer each request, a cheap quick one for easy questions and a bigger slower one for hard ones, to save money without hurting quality., OpenAI-compatible APIOpenAI-compatible API, n.: An AI service that accepts the same request format as OpenAI's, so software written for OpenAI's models can use other companies' or self-hosted models by changing an address and key., rate limitrate limit, n.: A cap on how much a customer may use an online AI service in a given time — so many requests or words per minute — beyond which extra requests are refused until later., LLM observabilityLLM observability, n.: Recording and inspecting what an AI application actually did on each request — prompts sent, documents found, tools used, answers given, time and cost — so problems can be found and fixed., LLMOpsLLMOps, n.: The everyday work of running chatbots and other products built on large language models: tracking prompts, testing answers, watching cost and quality, and updating safely when models change., routerrouter, n.: In a mixture-of-experts model, the small component that decides which specialist sub-networks should handle each word, like a dispatcher assigning jobs to workers., semantic cachingsemantic caching, n.: Saving an AI system's past answers and reusing one whenever a new question means the same as an old one, even if worded differently, to save time and money.