prefill

Inference

The phase of autoregressiveautoregressive model, n.: A model that produces a sequence one piece at a time, each new piece depending on the pieces already produced — like writing a sentence word by word without going back. generation in which a transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it. processes all prompt tokenstoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens. in one parallel forward passforward pass, n.: The step where a neural network takes an input and runs it through its layers, one after another, to produce an output — its answer or prediction., computing the keys and values that populate the KV cacheKV cache, n.: A memory store where a chatbot keeps its working notes on everything already read and written, so each new word does not require re-reading the whole conversation from scratch. and the logitslogit, n.: A raw score that a model gives each possible answer before the scores are turned into probabilities. In statistics, the 'log-odds': a probability rewritten on a scale running from minus to plus infinity. for the first output token. Because many tokens are processed together, prefill performs large matrix–matrix multiplications and is usually compute-bound, unlike the memory-boundmemory-bound, adj.: Describes a computer task slowed by how fast data can be fetched from memory rather than by how fast the chip can calculate — like a fast cook kept waiting for deliveries. decode phasedecode phase, n.: The second stage of an AI's reply, in which it writes the answer one word-piece at a time, each step depending on everything written so far.; its cost grows linearly with prompt length in the feed-forward layers and quadratically in attentionattention, n.: A way for an AI model to choose what to focus on. For each word it processes, the model scores how relevant every other word is and draws information mostly from the relevant ones.. Prefill is the main component of time to first tokentime to first token, n.: How long a chatbot takes to start answering: the pause between pressing enter and seeing the first word appear.. Serving systems avoid repeating it with prompt cachingprompt caching, n.: Saving the work a model did reading the start of a prompt, so later requests that begin the same way can skip that reading and come back faster and cheaper., split it into pieces with chunked prefillchunked prefill, n.: A serving trick that splits the reading of a long prompt into smaller pieces and slots them between other users' replies, so nobody's answer freezes while a huge document is read. so that it does not stall other users' decoding, or run it on separate hardware in disaggregated servingdisaggregated serving, n.: Running the two stages of an AI's reply, reading the prompt and writing the answer, on separate groups of chips, each tuned for its own job, and handing the work between them..

A 100,000-token prompt spends several seconds in prefill before the first word comes back.

Prompting

Text a developer places at the start of the model's own reply so that generation continues from it, for example an opening brace to force JSON or a fixed first sentence to set tone; also called response prefilling. Some APIs support it directly. Because it can also push a model past its refusals, it is studied as an attack surface; see prefill attackprefill attack, n.: Making a chatbot misbehave by writing the first words of its answer for it, so that it starts out sounding agreeable and carries on instead of refusing..