model routing
In plain wordsAutomatically choosing which AI model should answer each request, a cheap quick one for easy questions and a bigger slower one for hard ones, to save money without hurting quality.
The selection, for each incoming request, of which model or configuration should handle it, made before generation by a learned classifier, a small language modellarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code., hand-written rules or user signals. A router estimates a query's difficulty or domain and sends easy requests to small, fast, cheap models and hard ones to larger or reasoning modelsreasoning model, n.: An AI model trained to work through a problem step by step in hidden 'thinking' text before it answers. Letting it think longer generally makes it better at math, coding and planning., or picks a model specialized for code, a language or a modality; it may also choose a provider, a thinking budgetthinking budget, n.: A limit, set by whoever uses an AI model, on how much private step-by-step reasoning it may do before answering: more thinking usually means better answers but higher cost and slower replies. or whether to use tools. Routers are trained on preference or correctness data and judged by the cost–quality trade-off curve they achieve. Consumer assistants use routers to switch between fast and reasoning modes, and AI gatewaysAI gateway, n.: A checkpoint between an organization's apps and the AI models they use that handles access, costs, logging, safety checks and switching between providers in one place. offer routing as a service.
The router sends about 80 percent of support tickets to the small model and escalates the rest.
Origin
Builds on earlier work on selecting among predictors. LLM routing took off in 2023–2024 with research such as FrugalGPT (Lingjiao Chen, Matei Zaharia and James Zou, 2023), Hybrid LLM (2024) and LMSYS's RouteLLM (2024), and with routing startups. OpenAI's GPT-5 (August 2025) brought it to a mass audience with a real-time router choosing between fast and 'thinking' models.
Usage note
Distinct from the routerrouter, n.: In a mixture-of-experts model, the small component that decides which specialist sub-networks should handle each word, like a dispatcher assigning jobs to workers. inside a mixture of expertsmixture of experts, n.: A way to build very large AI models cheaply: the model contains many specialist sub-networks, and for each word only a few are switched on, picked by a small dispatcher network. layer, which assigns individual tokens to expert sub-networks within one model; model routing chooses between whole models. A model cascademodel cascade, n.: A chain of AI models ordered from cheapest to most capable: each request goes to the cheap one first and moves up the chain only if the answer does not look good enough. is the sequential variant that escalates only when a cheaper answer looks inadequate.
Related
- model cascademodel cascade, n.: A chain of AI models ordered from cheapest to most capable: each request goes to the cheap one first and moves up the chain only if the answer does not look good enough.
- inference costinference cost, n.: What it costs to use an AI model after it has been built: the computing power, electricity and hardware needed every time it answers, usually billed per chunk of text.
- AI gatewayAI gateway, n.: A checkpoint between an organization's apps and the AI models they use that handles access, costs, logging, safety checks and switching between providers in one place.
- reasoning modelreasoning model, n.: An AI model trained to work through a problem step by step in hidden 'thinking' text before it answers. Letting it think longer generally makes it better at math, coding and planning.
- small language modelsmall language model, n.: A language model built to be compact enough to run cheaply and quickly, even on a laptop or phone, trading some knowledge and skill for speed, privacy and lower cost.
- test-time computetest-time compute, n.: The computing effort an AI model spends while answering a question, as opposed to while being trained. Letting a model think longer or try many answers can make it markedly smarter.
- routerrouter, n.: In a mixture-of-experts model, the small component that decides which specialist sub-networks should handle each word, like a dispatcher assigning jobs to workers.
- mixture of expertsmixture of experts, n.: A way to build very large AI models cheaply: the model contains many specialist sub-networks, and for each word only a few are switched on, picked by a small dispatcher network.
Mentioned in
inference costinference cost, n.: What it costs to use an AI model after it has been built: the computing power, electricity and hardware needed every time it answers, usually billed per chunk of text., agentic workflowagentic workflow, n.: A multi-step process in which AI models draft, check, use tools and revise their work over several rounds, instead of answering in one shot; often the steps are laid out by developers in advance., small language modelsmall language model, n.: A language model built to be compact enough to run cheaply and quickly, even on a laptop or phone, trading some knowledge and skill for speed, privacy and lower cost., AI gatewayAI gateway, n.: A checkpoint between an organization's apps and the AI models they use that handles access, costs, logging, safety checks and switching between providers in one place., compound AI systemcompound AI system, n.: An AI product built from several working parts — models, search tools, code and checks — designed to work together, rather than a single model answering on its own., hybrid reasoning modelhybrid reasoning model, n.: A single AI model that can either answer right away or first think at length, with the user or developer choosing how much thinking to allow., model cascademodel cascade, n.: A chain of AI models ordered from cheapest to most capable: each request goes to the cheap one first and moves up the chain only if the answer does not look good enough., routerrouter, n.: In a mixture-of-experts model, the small component that decides which specialist sub-networks should handle each word, like a dispatcher assigning jobs to workers.