inference provider
In plain wordsA company that runs AI models on its own computers and sells access to them over the internet, charging by use, so customers need not buy or operate the hardware themselves.
A company or service that hosts models and sells access to them, usually through a pay-per-token model APImodel API, n.: A web service through which programs send requests to an AI model run by a company and receive its answers, usually paying per use instead of running the model themselves. or dedicated endpoints. The term covers model developers serving their own proprietary modelsproprietary model, n.: An AI model kept private by the company that made it; outsiders can use it only through the company's apps or paid online services and can never download it., cloud hyperscalers reselling many vendors' models, and specialist firms that serve open-weight modelsopen-weight model, n.: An AI model whose trained internal numbers are published for anyone to download and run on their own computers, even if the data and code used to make it stay private. on their own fleets of GPUsGPU, abbr.: A computer chip first designed to draw video-game graphics that turned out to be ideal for AI, because it can do thousands of simple calculations at the same time. or custom chips, competing on price, tokens per secondtokens per second, n.: A speed rating for AI text generation: how many word-pieces a model produces each second. Five to ten keeps up with a fast reader; hundreds feel instant., latencylatency, n.: How long you wait between asking an AI system for something and getting the answer. Lower latency means a snappier, more responsive system., context length and reliability. Aggregators sit in front of many providers behind a single OpenAI-compatible APIOpenAI-compatible API, n.: An AI service that accepts the same request format as OpenAI's, so software written for OpenAI's models can use other companies' or self-hosted models by changing an address and key.. The same open-weight model can behave differently across providers because of quantizationquantization, n.: Storing an AI model's numbers with fewer digits, like rounding prices to the nearest dollar, so the model takes less memory and runs faster, usually with only a small loss of accuracy., different serving engines, default sampling settings, context limits and bugs in tool-callingtool use, n.: An AI model's ability to reach beyond its own knowledge by operating outside tools — searching the web, running code, using a calculator or an app — and folding the results into its work. or chat-template handling, so careful evaluations name the provider as well as the model.
We send the open-weight model to whichever inference provider is cheapest that day, after checking its tool-call accuracy.
Origin
The phrase spread from 2023 as open-weight models such as Llama created a market for hosted inference separate from model development. In January 2025 Hugging Face branded the third-party serverless partners on its Hub 'Inference Providers', and aggregators such as OpenRouter list many providers per model.
Usage note
In the EU AI ActEU AI Act, prop. n.: The European Union's law on artificial intelligence, the first broad AI law anywhere: it bans some uses, sets strict checks for risky ones and puts duties on the makers of the biggest AI models., a 'provider' is whoever develops a system or model and places it on the market, which may not be the company hosting inference; see provider and deployerprovider and deployer, phr.: The two main roles in EU AI law: the provider is whoever makes an AI system and puts it out under its own name; the deployer is whoever uses it for work or official purposes..
Related
- model APImodel API, n.: A web service through which programs send requests to an AI model run by a company and receive its answers, usually paying per use instead of running the model themselves.
- model servingmodel serving, n.: Running a trained AI model as a live service, so that apps and people can send it requests and get answers back quickly, reliably and at a sensible cost.
- open-weight modelopen-weight model, n.: An AI model whose trained internal numbers are published for anyone to download and run on their own computers, even if the data and code used to make it stay private.
- inference costinference cost, n.: What it costs to use an AI model after it has been built: the computing power, electricity and hardware needed every time it answers, usually billed per chunk of text.
- AI gatewayAI gateway, n.: A checkpoint between an organization's apps and the AI models they use that handles access, costs, logging, safety checks and switching between providers in one place.
- OpenAI-compatible APIOpenAI-compatible API, n.: An AI service that accepts the same request format as OpenAI's, so software written for OpenAI's models can use other companies' or self-hosted models by changing an address and key.
- provider and deployerprovider and deployer, phr.: The two main roles in EU AI law: the provider is whoever makes an AI system and puts it out under its own name; the deployer is whoever uses it for work or official purposes.
Mentioned in
model APImodel API, n.: A web service through which programs send requests to an AI model run by a company and receive its answers, usually paying per use instead of running the model themselves., on-device AIon-device AI, n.: AI that runs directly on your own phone, laptop, car or headset rather than on a company's distant servers, so it can work offline, respond instantly and keep your data on the device., model servingmodel serving, n.: Running a trained AI model as a live service, so that apps and people can send it requests and get answers back quickly, reliably and at a sensible cost., OllamaOllama, prop. n.: A free app that makes it easy to download and run AI language models on your own computer with a single command, keeping your data on your machine., OpenAI-compatible APIOpenAI-compatible API, n.: An AI service that accepts the same request format as OpenAI's, so software written for OpenAI's models can use other companies' or self-hosted models by changing an address and key., rate limitrate limit, n.: A cap on how much a customer may use an online AI service in a given time — so many requests or words per minute — beyond which extra requests are refused until later., vLLMvLLM, prop. n.: A free, open-source program for running large language models on servers efficiently, so one machine can answer many more users at once. Many companies use it to host AI models.