model API
In plain wordsA web service through which programs send requests to an AI model run by a company and receive its answers, usually paying per use instead of running the model themselves.
An application programming interface through which client software submits inputs to a hosted AI model over the network — usually HTTPS requests with JSON bodies, authenticated by an API key — and receives outputs, either complete or streamedstreaming, n.: Sending an AI's answer piece by piece while it is still being written, so the words appear on screen one after another instead of all at once at the end.. For language modelslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code., a request carries a list of messages, a system promptsystem prompt, n.: Background instructions given to a chatbot before the conversation starts, setting its role, rules, tone and tools. Users usually never see it, but it shapes every reply., optional tooltool, n.: Any outside capability an AI model is allowed to use — a web search, a calculator, a code runner, a database or an app — described to the model so it can ask for it by name. definitions, a schema for structured outputsstructured outputs, n.: Getting an AI model to answer in an exact, machine-readable format — like a form with fixed fields — so other software can use the answer without guesswork., and sampling settings such as temperaturetemperature, n.: A setting that controls how adventurous an AI model's word choices are. Low temperature makes it pick the safest, most likely words; high temperature makes it more varied and surprising, but more error-prone. and maximum output length; the response returns generated content, tool callstool call, n.: A single request from an AI model to use one of its tools — naming the tool and filling in the details — which the surrounding software then carries out and reports back on. and usage counts. Pricing is typically per million input and output tokenstoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens., with discounts for cachedprompt caching, n.: Saving the work a model did reading the start of a prompt, so later requests that begin the same way can skip that reading and come back faster and cheaper. prompts and batchbatch inference, n.: Running an AI model over a big pile of inputs as one job rather than answering each request live, like processing a stack of forms overnight. Results come back later but cost less. jobs, and access is governed by rate limitsrate limit, n.: A cap on how much a customer may use an online AI service in a given time — so many requests or words per minute — beyond which extra requests are refused until later., usage tiers and terms of service. Model APIs let developers use frontier modelsfrontier model, n.: One of the most capable AI models in existence at a given time, built by a handful of leading labs at great cost, and the main target of AI safety rules. without owning hardware, and let providers keep weightsweights, n.: The learned numbers inside a neural network that set how strongly each part influences the next. The finished set of weights is, in effect, the trained model itself. private, update models and enforce usage policies; in exchange, customers accept vendor dependence, data leaving their systems and model versions that can be retired.
We prototyped on a model API and only considered self-hosting once monthly spend passed five figures.
Origin
Hosted prediction APIs date from the early 2010s, notably the Google Prediction API (2010) and cloud vision and speech services. OpenAIOpenAI, prop. n.: The American company that makes ChatGPT and the GPT models. It began in 2015 as a nonprofit research lab and is now a for-profit company overseen by a nonprofit foundation.'s GPT-3GPT-3, prop. n.: A huge 2020 OpenAI language model that could perform new tasks simply from instructions or a few examples written into its prompt, without retraining. It made prompting a new way of programming. API (June 2020) set the pattern of selling access to a large model whose weights stay private, and its Chat Completions API (March 2023) became an interface many others copied.
Usage note
Also called closed or API-only access, in contrast with releasing open weightsopen-weight model, n.: An AI model whose trained internal numbers are published for anyone to download and run on their own computers, even if the data and code used to make it stay private.. Many providers also expose an OpenAI-compatible APIOpenAI-compatible API, n.: An AI service that accepts the same request format as OpenAI's, so software written for OpenAI's models can use other companies' or self-hosted models by changing an address and key., so client code can switch vendors by changing a URL and key.
Related
- OpenAI-compatible APIOpenAI-compatible API, n.: An AI service that accepts the same request format as OpenAI's, so software written for OpenAI's models can use other companies' or self-hosted models by changing an address and key.
- rate limitrate limit, n.: A cap on how much a customer may use an online AI service in a given time — so many requests or words per minute — beyond which extra requests are refused until later.
- AI gatewayAI gateway, n.: A checkpoint between an organization's apps and the AI models they use that handles access, costs, logging, safety checks and switching between providers in one place.
- proprietary modelproprietary model, n.: An AI model kept private by the company that made it; outsiders can use it only through the company's apps or paid online services and can never download it.
- inference providerinference provider, n.: A company that runs AI models on its own computers and sells access to them over the internet, charging by use, so customers need not buy or operate the hardware themselves.
- structured outputsstructured outputs, n.: Getting an AI model to answer in an exact, machine-readable format — like a form with fixed fields — so other software can use the answer without guesswork.
- prompt cachingprompt caching, n.: Saving the work a model did reading the start of a prompt, so later requests that begin the same way can skip that reading and come back faster and cheaper.
- open-weight modelopen-weight model, n.: An AI model whose trained internal numbers are published for anyone to download and run on their own computers, even if the data and code used to make it stay private.
Mentioned in
proprietary modelproprietary model, n.: An AI model kept private by the company that made it; outsiders can use it only through the company's apps or paid online services and can never download it., AI applicationAI application, n.: A program or product that uses AI to do something useful for people, such as a translation app, a medical-scan checker or a coding assistant., AI engineerAI engineer, n.: A software developer who builds products using ready-made AI models — connecting them to data and tools and testing that they work well — rather than inventing or training the models., AI engineeringAI engineering, n.: The craft of building useful, reliable products on top of AI models — choosing models, writing prompts, connecting data and tools, testing quality and running it all safely at scale., AI modelAI model, n.: The trained core of an AI: a large file of learned numbers plus the recipe for using them, which turns an input such as a question or a photo into an output., AI stackAI stack, n.: The layers of technology an AI product depends on, from chips and data centers at the bottom, through the models, up to the apps people use, each built on the layer below., batch inferencebatch inference, n.: Running an AI model over a big pile of inputs as one job rather than answering each request live, like processing a stack of forms overnight. Results come back later but cost less., completioncompletion, n.: The text an AI language model writes in response to a prompt — literally, its continuation of the text it was given., inference providerinference provider, n.: A company that runs AI models on its own computers and sells access to them over the internet, charging by use, so customers need not buy or operate the hardware themselves., OpenAI-compatible APIOpenAI-compatible API, n.: An AI service that accepts the same request format as OpenAI's, so software written for OpenAI's models can use other companies' or self-hosted models by changing an address and key., parameterparameter, n.: One of the adjustable numbers inside an AI model that are tuned during training. Large language models have billions of them, and together they store what the model has learned., prompt cachingprompt caching, n.: Saving the work a model did reading the start of a prompt, so later requests that begin the same way can skip that reading and come back faster and cheaper., rate limitrate limit, n.: A cap on how much a customer may use an online AI service in a given time — so many requests or words per minute — beyond which extra requests are refused until later., staged deploymentstaged deployment, n.: Releasing a new AI system step by step — first to testers, then to limited groups, then widely — so that problems can be spotted and fixed before everyone has it., streamingstreaming, n.: Sending an AI's answer piece by piece while it is still being written, so the words appear on screen one after another instead of all at once at the end., structured outputsstructured outputs, n.: Getting an AI model to answer in an exact, machine-readable format — like a form with fixed fields — so other software can use the answer without guesswork., wrapperwrapper, n.: A product built mostly on top of someone else's AI model, adding a simple interface or a few instructions; often used as a put-down implying the product is easy to copy., AI gatewayAI gateway, n.: A checkpoint between an organization's apps and the AI models they use that handles access, costs, logging, safety checks and switching between providers in one place., forward deployed engineerforward deployed engineer, n.: An engineer from an AI or software company who works directly inside a customer's organization, building and tailoring the company's systems to that customer's real problems., JSON modeJSON mode, n.: A setting on some AI services that makes the model reply only in JSON, a standard text format for data, so that programs can read the reply reliably., LLMOpsLLMOps, n.: The everyday work of running chatbots and other products built on large language models: tracking prompts, testing answers, watching cost and quality, and updating safely when models change., model deprecationmodel deprecation, n.: When an AI company announces that an older model will stop being offered and then switches it off, so the apps and people relying on it must move to a newer one., structured accessstructured access, n.: Letting people use an AI system through a controlled channel, such as a website or programming interface, instead of handing over the model itself, so that the developer can limit misuse.