inference cost
In plain wordsWhat it costs to use an AI model after it has been built: the computing power, electricity and hardware needed every time it answers, usually billed per chunk of text.
The resources consumed in running a trained model (accelerator time, energy, memory and networking) and the price charged for them, as distinct from the one-off cost of trainingtraining, n.: The process by which an AI model learns: it is shown huge numbers of examples and its internal settings are adjusted, bit by bit, until its answers get better.. For language modelslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. it is usually quoted per million input and output tokenstoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens., with output priced several times higher because decodingdecode phase, n.: The second stage of an AI's reply, in which it writes the answer one word-piece at a time, each step depending on everything written so far. uses hardware less efficiently than prefillprefill, n.: The first stage of an AI's reply, in which it reads your whole prompt in one go before it starts writing. Long prompts make this stage take longer.; cached input (prompt cachingprompt caching, n.: Saving the work a model did reading the start of a prompt, so later requests that begin the same way can skip that reading and come back faster and cheaper.) and batch inferencebatch inference, n.: Running an AI model over a big pile of inputs as one job rather than answering each request live, like processing a stack of forms overnight. Results come back later but cost less. are discounted. Underlying cost scales with active parametersactive parameters, n.: The part of a model's internal numbers actually used to process each word. Some big models use only a small slice of themselves at a time, making them cheaper to run than their size suggests. (about two floating-point operations per active parameter per token), context length, hardware utilization and batch sizebatch size, n.: How many training examples a model looks at before each adjustment of its settings. Bigger batches use hardware efficiently but bring diminishing returns; smaller ones are noisier but cheaper per step., and is cut by quantizationquantization, n.: Storing an AI model's numbers with fewer digits, like rounding prices to the nearest dollar, so the model takes less memory and runs faster, usually with only a small loss of accuracy., speculative decodingspeculative decoding, n.: A speed-up trick in which a small, fast model guesses the next few words and the big model checks all the guesses at once, keeping the right ones: faster answers, same quality., distillationknowledge distillation, n.: Training a smaller or cheaper 'student' AI to copy the behavior of a larger 'teacher' AI, so the student keeps much of the teacher's skill while being faster to run. into smaller models, model routingmodel routing, n.: Automatically choosing which AI model should answer each request, a cheap quick one for easy questions and a bigger slower one for hard ones, to save money without hurting quality., and better kernels and chips. Prices for a fixed level of capability have fallen roughly tenfold or more per year since 2021, yet reasoning modelsreasoning model, n.: An AI model trained to work through a problem step by step in hidden 'thinking' text before it answers. Letting it think longer generally makes it better at math, coding and planning. and agentsAI agent, n.: An AI system that does not just answer but acts: given a goal, it decides on steps, uses tools such as a browser or code editor, checks the results and keeps going until the job is done. consume far more tokens per task, so total inference spending keeps growing.
The agent costs about 40 cents per task because each run reads two million tokens.
Origin
Discussed since deep learning entered products in the 2010s, the term became central after ChatGPT (2022), when serving costs shaped pricing and chip demand. Guido Appenzeller of Andreessen Horowitz introduced the name 'LLMflation' in November 2024 for the roughly tenfold annual fall in cost at constant capability: GPT-3-level performance went from $60 to $0.06 per million tokens in three years.
Usage note
Cheaper tokens have not meant smaller bills: as in the Jevons paradox of energy economics, falling unit cost has encouraged far greater use, especially by reasoning models and agents. Training cost is paid once per model; inference cost is paid with every use.
Related
- tokens per secondtokens per second, n.: A speed rating for AI text generation: how many word-pieces a model produces each second. Five to ten keeps up with a fast reader; hundreds feel instant.
- throughputthroughput, n.: How much work an AI system gets done per second across all its users at once — for a chatbot service, roughly how much text it can produce in total each second.
- active parametersactive parameters, n.: The part of a model's internal numbers actually used to process each word. Some big models use only a small slice of themselves at a time, making them cheaper to run than their size suggests.
- prompt cachingprompt caching, n.: Saving the work a model did reading the start of a prompt, so later requests that begin the same way can skip that reading and come back faster and cheaper.
- batch inferencebatch inference, n.: Running an AI model over a big pile of inputs as one job rather than answering each request live, like processing a stack of forms overnight. Results come back later but cost less.
- model routingmodel routing, n.: Automatically choosing which AI model should answer each request, a cheap quick one for easy questions and a bigger slower one for hard ones, to save money without hurting quality.
- environmental impact of AIenvironmental impact of AI, n.: The effects of AI on the planet: the electricity, water, materials and land used to make chips, build data centers and train and run models, and the pollution that results.
- training computetraining compute, n.: The total number of arithmetic operations performed to train a model. It is the standard yardstick for the size of an AI training effort, and some laws use it to decide which models face extra rules.
Sources
- Appenzeller, G. (2024). Welcome to LLMflation – LLM inference cost is going down fast. Andreessen Horowitz, 12 November 2024. a16z.com/llmflation-llm-inference-cost
Mentioned in
on-device AIon-device AI, n.: AI that runs directly on your own phone, laptop, car or headset rather than on a company's distant servers, so it can work offline, respond instantly and keep your data on the device., active parametersactive parameters, n.: The part of a model's internal numbers actually used to process each word. Some big models use only a small slice of themselves at a time, making them cheaper to run than their size suggests., batch inferencebatch inference, n.: Running an AI model over a big pile of inputs as one job rather than answering each request live, like processing a stack of forms overnight. Results come back later but cost less., GPU-hourGPU-hour, n.: One graphics chip working for one hour — the everyday unit for how much computing a job used and for renting AI chips by the hour., inferenceinference, n.: Drawing a conclusion from evidence. In today's AI it most often means running an already-trained model to get an answer, as opposed to training it., inference providerinference provider, n.: A company that runs AI models on its own computers and sells access to them over the internet, charging by use, so customers need not buy or operate the hardware themselves., Jevons paradoxJevons paradox, n.: The observation that making something more efficient can make people use more of it overall because it gets cheaper — as with coal in the 1800s, and with AI computing today., model compressionmodel compression, n.: The family of techniques for making AI models smaller, faster or cheaper to run while keeping almost all of their accuracy., model routingmodel routing, n.: Automatically choosing which AI model should answer each request, a cheap quick one for easy questions and a bigger slower one for hard ones, to save money without hurting quality., overtrainingovertraining, n.: Deliberately training a model on much more data than would be most efficient for its training budget, so that a smaller model reaches good quality and is cheaper to run afterwards., prompt cachingprompt caching, n.: Saving the work a model did reading the start of a prompt, so later requests that begin the same way can skip that reading and come back faster and cheaper., small language modelsmall language model, n.: A language model built to be compact enough to run cheaply and quickly, even on a laptop or phone, trading some knowledge and skill for speed, privacy and lower cost., test-time computetest-time compute, n.: The computing effort an AI model spends while answering a question, as opposed to while being trained. Letting a model think longer or try many answers can make it markedly smarter., thinking budgetthinking budget, n.: A limit, set by whoever uses an AI model, on how much private step-by-step reasoning it may do before answering: more thinking usually means better answers but higher cost and slower replies., throughputthroughput, n.: How much work an AI system gets done per second across all its users at once — for a chatbot service, roughly how much text it can produce in total each second., early exitearly exit, n.: Letting an AI model stop partway through its layers and answer early when it is already confident, saving computing effort on easy inputs., hybrid reasoning modelhybrid reasoning model, n.: A single AI model that can either answer right away or first think at length, with the user or developer choosing how much thinking to allow., model cascademodel cascade, n.: A chain of AI models ordered from cheapest to most capable: each request goes to the cheap one first and moves up the chain only if the answer does not look good enough., overthinkingoverthinking, n.: When an AI reasoning model spends far more thinking than a problem needs — pages of deliberation for a simple sum — wasting time and money and sometimes talking itself out of a right answer., semantic cachingsemantic caching, n.: Saving an AI system's past answers and reusing one whenever a new question means the same as an old one, even if worded differently, to save time and money., sleep-time computesleep-time compute, n.: Letting an AI think about your documents or project while it would otherwise sit idle, before you ask anything, so it can answer later questions faster and more cheaply.