rate limit
In plain wordsA cap on how much a customer may use an online AI service in a given time — so many requests or words per minute — beyond which extra requests are refused until later.
A limit imposed by a service on the volume of requests a client may make in a time window, enforced by rejecting or delaying the excess. Model APIsmodel API, n.: A web service through which programs send requests to an AI model run by a company and receive its answers, usually paying per use instead of running the model themselves. typically meter several dimensions at once — requests per minute (RPM), input and output tokenstoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens. per minute (TPM), tokens per day and concurrent requests — set per account, project, model and usage tier, with tiers rising as spending history grows. Exceeding a limit returns an HTTP 429 'Too Many Requests' error, often with headers giving the remaining allowance and the time until reset. Limits protect shared accelerator capacity from overload and abuse and ration it among customers; providers implement them with token-bucket or sliding-window algorithms. Clients cope by retrying with exponential backoff and random jitter, queuing and batching work, using batchbatch inference, n.: Running an AI model over a big pile of inputs as one job rather than answering each request live, like processing a stack of forms overnight. Results come back later but cost less. endpoints, spreading load across regions or providers through an AI gatewayAI gateway, n.: A checkpoint between an organization's apps and the AI models they use that handles access, costs, logging, safety checks and switching between providers in one place., or buying reserved capacity.
The eval run kept hitting the rate limit, so we added exponential backoff and halved concurrency.
Origin
Rate limiting is long-standing networking and web-API practice, applied to public web APIs since the 2000s. Limits counted in model tokens are specific to language-model services and became a routine constraint for developers with the GPT-3GPT-3, prop. n.: A huge 2020 OpenAI language model that could perform new tasks simply from instructions or a few examples written into its prompt, without retraining. It made prompting a new way of programming. and ChatGPTChatGPT, prop. n.: OpenAI's AI chatbot, released free to the public in November 2022, which answers questions, writes and converses in everyday language. Its sudden popularity began the current AI boom. APIs of 2020–2023, when shortages of GPUs made serving capacity scarce.
Usage note
The 'token' in tokens per minute is a unit of model text, unrelated to the 'token bucket' of the limiting algorithm. A rate limit differs from a spending cap and from the context windowcontext window, n.: How much an AI model can read and keep in view at once: the instructions, the conversation so far and any attached documents. Text that does not fit has to be cut, summarized or left out., which bounds the size of a single request.
Related
- model APImodel API, n.: A web service through which programs send requests to an AI model run by a company and receive its answers, usually paying per use instead of running the model themselves.
- AI gatewayAI gateway, n.: A checkpoint between an organization's apps and the AI models they use that handles access, costs, logging, safety checks and switching between providers in one place.
- batch inferencebatch inference, n.: Running an AI model over a big pile of inputs as one job rather than answering each request live, like processing a stack of forms overnight. Results come back later but cost less.
- service-level objectiveservice-level objective, n.: A specific, measurable target for how well a service should perform — for example, 99.9 percent of requests answered within two seconds — used to decide when engineers must act.
- throughputthroughput, n.: How much work an AI system gets done per second across all its users at once — for a chatbot service, roughly how much text it can produce in total each second.
- inference providerinference provider, n.: A company that runs AI models on its own computers and sells access to them over the internet, charging by use, so customers need not buy or operate the hardware themselves.
Mentioned in
model APImodel API, n.: A web service through which programs send requests to an AI model run by a company and receive its answers, usually paying per use instead of running the model themselves., proprietary modelproprietary model, n.: An AI model kept private by the company that made it; outsiders can use it only through the company's apps or paid online services and can never download it., model extractionmodel extraction, n.: Copying an AI model without permission by asking it huge numbers of questions and using the answers to build a look-alike, or to work out hidden details of how it was made., AI gatewayAI gateway, n.: A checkpoint between an organization's apps and the AI models they use that handles access, costs, logging, safety checks and switching between providers in one place., service-level objectiveservice-level objective, n.: A specific, measurable target for how well a service should perform — for example, 99.9 percent of requests answered within two seconds — used to decide when engineers must act., sponge examplesponge example, n.: An input crafted to make an AI system work as hard as possible — burning extra energy and time — to slow it down or run up costs, rather than to make it answer wrongly.