inference
In plain wordsDrawing a conclusion from evidence. In today's AI it most often means running an already-trained model to get an answer, as opposed to training it.
- 1
Machine learning (deployment)
The use of a trained model to compute outputs for new inputs while its weightsweights, n.: The learned numbers inside a neural network that set how strongly each part influences the next. The finished set of weights is, in effect, the trained model itself. stay fixed: answering a prompt, classifying an image, generating a picture. For language modelslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code., inference consists of a prefillprefill, n.: The first stage of an AI's reply, in which it reads your whole prompt in one go before it starts writing. Long prompts make this stage take longer. phase that processes the prompt and a decode phasedecode phase, n.: The second stage of an AI's reply, in which it writes the answer one word-piece at a time, each step depending on everything written so far. that generates tokens one at a time with the help of a KV cacheKV cache, n.: A memory store where a chatbot keeps its working notes on everything already read and written, so each new word does not require re-reading the whole conversation from scratch.. Its economics (latencylatency, n.: How long you wait between asking an AI system for something and getting the answer. Lower latency means a snappier, more responsive system., throughputthroughput, n.: How much work an AI system gets done per second across all its users at once — for a chatbot service, roughly how much text it can produce in total each second. and inference costinference cost, n.: What it costs to use an AI model after it has been built: the computing power, electricity and hardware needed every time it answers, usually billed per chunk of text.) are the concern of model servingmodel serving, n.: Running a trained AI model as a live service, so that apps and people can send it requests and get answers back quickly, reliably and at a sensible cost.. Spending more computation at inference, for example by letting a model reason longer before answering, is called test-time computetest-time compute, n.: The computing effort an AI model spends while answering a question, as opposed to while being trained. Letting a model think longer or try many answers can make it markedly smarter..
Training was a one-off cost; inference is what shows up on the monthly bill.
- 2
Statistics
Drawing conclusions about a population or data-generating process from a sample: estimating parameters, quantifying uncertainty with confidence intervalsconfidence interval, n.: A range of plausible values for something being estimated, such as 'accuracy 82 percent, give or take 2 points', reflecting the uncertainty that comes from measuring only a sample. or posterior distributions, and testing hypotheses. Frequentist and Bayesian inferenceBayesian inference, n.: Reasoning that treats unknowns as uncertain beliefs and updates them with evidence using Bayes' theorem. The result is not one answer but a range of answers, each with a probability. are the two main schools; approximate inference, such as variational methods or Markov chain Monte CarloMarkov chain Monte Carlo, n.: A family of methods for drawing random examples from a complicated probability distribution by taking a long, carefully designed random walk that eventually visits each region in proportion to its probability., computes posteriors that cannot be found exactly.
- 3
Logic and symbolic AI
The derivation of conclusions from premises by rules of reasoning: deduction, induction or abduction. In expert systemsexpert system, n.: A program that stores a human specialist's know-how as many if–then rules and uses them to give advice or make decisions in a narrow field, such as diagnosing infections or configuring computers., an inference engineinference engine, n.: The part of an AI system that does the working-out: in older AI, the program that applies rules to facts; in today's AI, the software that runs a trained model to produce answers. applies rules to a knowledge baseknowledge base, n.: A store of facts and rules that a program can reason with: like a database, but organized so that new conclusions can be worked out from it, not just looked up. by forward chainingforward chaining, n.: A way of reasoning that starts from known facts and keeps applying rules to them, adding each new conclusion, until it reaches an answer or nothing new can be concluded. or backward chainingbackward chaining, n.: A way of reasoning that starts from the question ('is this true?') and works backward, looking for rules that would prove it and then checking whether their conditions hold. to reach new facts or decisions.
- 4
Natural language processing
Short for natural language inference, also called recognizing textual entailment: the task of deciding whether a premise sentence entails, contradicts or is neutral toward a hypothesis sentence, long used as a test of language understanding in benchmarksbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time. such as GLUEGLUE, prop. n.: A 2018 bundle of nine English language-understanding tests combined into one score, which became the yardstick for the first wave of pretrained language models such as BERT..
Origin
From Latin inferre, 'to bring in; to conclude'. The logical and statistical senses are centuries old. The deployment sense spread in deep-learning engineering in the mid-2010s, as hardware and software were built specifically for running trained networks rather than training them; Google's first TPU, announced in 2016, was an inference-only chip.
Usage note
The deployment sense confuses statisticians, for whom inference means estimating unknowns, not running a fixed predictor. The EU AI Act uses 'infer' broadly: a system's capability to infer how to generate outputs from inputs is the hallmark of an AI systemAI system, n.: A complete piece of technology that uses AI to do something, such as screening loan applications, driving a car or chatting with customers, including the AI model and the software around it. (Recital 12).
Related
- model servingmodel serving, n.: Running a trained AI model as a live service, so that apps and people can send it requests and get answers back quickly, reliably and at a sensible cost.
- test-time computetest-time compute, n.: The computing effort an AI model spends while answering a question, as opposed to while being trained. Letting a model think longer or try many answers can make it markedly smarter.
- Bayesian inferenceBayesian inference, n.: Reasoning that treats unknowns as uncertain beliefs and updates them with evidence using Bayes' theorem. The result is not one answer but a range of answers, each with a probability.
- inference engineinference engine, n.: The part of an AI system that does the working-out: in older AI, the program that applies rules to facts; in today's AI, the software that runs a trained model to produce answers.
- inference costinference cost, n.: What it costs to use an AI model after it has been built: the computing power, electricity and hardware needed every time it answers, usually billed per chunk of text.
- trainingtraining, n.: The process by which an AI model learns: it is shown huge numbers of examples and its internal settings are adjusted, bit by bit, until its answers get better.
- predictionprediction, n.: A model's output for a given input: its best guess at an unknown answer. In AI it need not be about the future; labeling a photo 'cat' counts as a prediction.
- trainingtraining, n.: The process by which an AI model learns: it is shown huge numbers of examples and its internal settings are adjusted, bit by bit, until its answers get better.
Mentioned in
environmental impact of AIenvironmental impact of AI, n.: The effects of AI on the planet: the electricity, water, materials and land used to make chips, build data centers and train and run models, and the pollution that results., modelmodel, n.: A simplified working version of something. In AI it usually means the trained program, a big set of learned numbers, that takes an input and produces an answer, prediction or piece of content., trainingtraining, n.: The process by which an AI model learns: it is shown huge numbers of examples and its internal settings are adjusted, bit by bit, until its answers get better., AI modelAI model, n.: The trained core of an AI: a large file of learned numbers plus the recipe for using them, which turns an input such as a question or a photo into an output., AI supercomputerAI supercomputer, n.: A huge network of AI chips — often tens or hundreds of thousands — wired together in a data center to train the largest AI models., AMDAMD, prop. n.: An American chip company that makes computer processors and graphics chips, and Nvidia's main rival in the chips used for artificial intelligence., CPUCPU, abbr.: The main general-purpose chip in a computer, good at running many different programs step by step. In AI systems it usually coordinates the work while specialized chips do the heavy math., forward passforward pass, n.: The step where a neural network takes an input and runs it through its layers, one after another, to produce an output — its answer or prediction., FP8FP8, abbr.: An 8-bit number format that uses a quarter of the memory of the standard 32-bit format, letting new AI chips train and run models roughly twice as fast as with 16 bits., high-bandwidth memoryhigh-bandwidth memory, n.: Very fast memory made by stacking memory chips on top of each other and placing them right next to the processor, so AI chips can be fed data quickly enough., inference engineinference engine, n.: The part of an AI system that does the working-out: in older AI, the program that applies rules to facts; in today's AI, the software that runs a trained model to produce answers., learninglearning, n.: Getting better at something through experience. For a machine, it means changing its internal settings based on data so that it performs a task better than before., model servingmodel serving, n.: Running a trained AI model as a live service, so that apps and people can send it requests and get answers back quickly, reliably and at a sensible cost., NPUNPU, abbr.: A small AI chip, or part of a chip, built into phones, laptops and other devices so they can run AI features quickly without draining the battery or using the cloud., predictionprediction, n.: A model's output for a given input: its best guess at an unknown answer. In AI it need not be about the future; labeling a photo 'cat' counts as a prediction., reasoningreasoning, n.: Working out conclusions step by step from facts, rules or evidence, as in solving a math problem or planning a trip. Whether today's AI truly reasons, or imitates reasoning, is debated., structured outputsstructured outputs, n.: Getting an AI model to answer in an exact, machine-readable format — like a form with fixed fields — so other software can use the answer without guesswork., test-time computetest-time compute, n.: The computing effort an AI model spends while answering a question, as opposed to while being trained. Letting a model think longer or try many answers can make it markedly smarter., augmented language modelaugmented language model, n.: A language model hooked up to extras — a search engine, a calculator, a memory store or other tools — so it can do things a model working only from its training cannot., CerebrasCerebras, prop. n.: An American chip company that makes a single giant processor the size of a whole silicon wafer, used to train and run AI models very quickly., CoreWeaveCoreWeave, prop. n.: An American company that rents out huge numbers of Nvidia graphics processors over the internet to AI developers such as Microsoft, OpenAI and Meta., feature storefeature store, n.: A shared library of ready-made input signals for models — such as a customer's spending last week — computed one consistent way and supplied both when models are trained and when they run live., FP4FP4, abbr.: A tiny 4-bit number format that can represent only sixteen different values. With clever scaling tricks it lets the newest AI chips run, and even train, models faster and with less memory., MLCommonsMLCommons, prop. n.: An industry and university consortium that runs MLPerf, the standard tests of how fast computers train and run AI models, as well as benchmarks of AI safety.