throughput
In plain wordsHow much work an AI system gets done per second across all its users at once — for a chatbot service, roughly how much text it can produce in total each second.
The rate at which a system completes work, measured in aggregate over all concurrent requests: requests or queries per second for classifiers and search, and output tokens per second summed across users for language-modellarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. serving. Throughput rises with batch sizebatch size, n.: How many training examples a model looks at before each adjustment of its settings. Bigger batches use hardware efficiently but bring diminishing returns; smaller ones are noisier but cheaper per step. because each read of the weightsweights, n.: The learned numbers inside a neural network that set how strongly each part influences the next. The finished set of weights is, in effect, the trained model itself. from memory is shared by more sequences, until the hardware becomes compute-limited or the KV cacheKV cache, n.: A memory store where a chatbot keeps its working notes on everything already read and written, so each new word does not require re-reading the whole conversation from scratch. fills memory; it is therefore the main driver of cost per token. It must be read together with latencylatency, n.: How long you wait between asking an AI system for something and getting the answer. Lower latency means a snappier, more responsive system., since a server can post high throughput while each user waits longer. Serving research therefore reports throughput at a fixed latency target, or goodputgoodput, n.: The share of a big training job's time and computing that actually moves training forward, after subtracting time lost to crashes, restarts, waiting and repeated work. When serving users, it means the rate of requests answered fast enough., the rate of requests completed within their service-level objectives. In training, throughput means samples or tokens processed per second.
Doubling the batch size raised throughput by 70 percent but pushed time to first token past our target.
Origin
An industrial-engineering word for a process's output per unit of time, borrowed early into computing for jobs or transactions completed. Serving papers such as Orca (2022), which reported a 36.9-fold gain 'at the same level of latency', and vLLM (2023) made latency-matched throughput the standard way to compare LLM inference systems.
Usage note
Per-user generation speed is also quoted in tokens per secondtokens per second, n.: A speed rating for AI text generation: how many word-pieces a model produces each second. Five to ten keeps up with a fast reader; hundreds feel instant.; always check whether a figure is per request or aggregate across the server.
Related
- latencylatency, n.: How long you wait between asking an AI system for something and getting the answer. Lower latency means a snappier, more responsive system.
- tokens per secondtokens per second, n.: A speed rating for AI text generation: how many word-pieces a model produces each second. Five to ten keeps up with a fast reader; hundreds feel instant.
- goodputgoodput, n.: The share of a big training job's time and computing that actually moves training forward, after subtracting time lost to crashes, restarts, waiting and repeated work. When serving users, it means the rate of requests answered fast enough.
- continuous batchingcontinuous batching, n.: A way for an AI server to handle many conversations together, letting new ones join and finished ones leave after every word, instead of waiting for a whole group to finish.
- batch sizebatch size, n.: How many training examples a model looks at before each adjustment of its settings. Bigger batches use hardware efficiently but bring diminishing returns; smaller ones are noisier but cheaper per step.
- inference costinference cost, n.: What it costs to use an AI model after it has been built: the computing power, electricity and hardware needed every time it answers, usually billed per chunk of text.
- latencylatency, n.: How long you wait between asking an AI system for something and getting the answer. Lower latency means a snappier, more responsive system.
Sources
- Yu, G.-I., Jeong, J. S., Kim, G.-W., Kim, S. & Chun, B.-G. (2022). Orca: A Distributed Serving System for Transformer-Based Generative Models. OSDI 2022. usenix.org/conference/osdi22/presentation/yu
Mentioned in
inference costinference cost, n.: What it costs to use an AI model after it has been built: the computing power, electricity and hardware needed every time it answers, usually billed per chunk of text., batch inferencebatch inference, n.: Running an AI model over a big pile of inputs as one job rather than answering each request live, like processing a stack of forms overnight. Results come back later but cost less., continuous batchingcontinuous batching, n.: A way for an AI server to handle many conversations together, letting new ones join and finished ones leave after every word, instead of waiting for a whole group to finish., inferenceinference, n.: Drawing a conclusion from evidence. In today's AI it most often means running an already-trained model to get an answer, as opposed to training it., inference engineinference engine, n.: The part of an AI system that does the working-out: in older AI, the program that applies rules to facts; in today's AI, the software that runs a trained model to produce answers., latencylatency, n.: How long you wait between asking an AI system for something and getting the answer. Lower latency means a snappier, more responsive system., model monitoringmodel monitoring, n.: Continuously checking an AI model after it goes live — what data it receives, what it predicts and whether it is still right — so that problems are caught before they cause harm., model servingmodel serving, n.: Running a trained AI model as a live service, so that apps and people can send it requests and get answers back quickly, reliably and at a sensible cost., rate limitrate limit, n.: A cap on how much a customer may use an online AI service in a given time — so many requests or words per minute — beyond which extra requests are refused until later., tokens per secondtokens per second, n.: A speed rating for AI text generation: how many word-pieces a model produces each second. Five to ten keeps up with a fast reader; hundreds feel instant., goodputgoodput, n.: The share of a big training job's time and computing that actually moves training forward, after subtracting time lost to crashes, restarts, waiting and repeated work. When serving users, it means the rate of requests answered fast enough., MLPerfMLPerf, prop. n.: A standard set of speed tests for AI hardware and software, where chip makers and cloud companies compare how fast their systems train and run AI models., PagedAttentionPagedAttention, prop. n.: A way of storing an AI model's working memory in small fixed-size blocks, like the pages of a computer's memory, so little space is wasted and more users can be served at once., service-level objectiveservice-level objective, n.: A specific, measurable target for how well a service should perform — for example, 99.9 percent of requests answered within two seconds — used to decide when engineers must act.