decoding
In plain wordsThe method a language model uses to turn its predictions into actual text — for example, always picking the most likely next word, or choosing somewhat at random among the likely ones.
- 1
Language models
The procedure for producing an output sequence from a model's predicted next-token distributions. At each step the model yields logitslogit, n.: A raw score that a model gives each possible answer before the scores are turned into probabilities. In statistics, the 'log-odds': a probability rewritten on a scale running from minus to plus infinity. over the vocabularyvocabulary, n.: The fixed list of all the tokens a language model knows, each with its own number. Everything the model reads or writes must be built from pieces on this list.; a decoding strategy transforms them — applying temperaturetemperature, n.: A setting that controls how adventurous an AI model's word choices are. Low temperature makes it pick the safest, most likely words; high temperature makes it more varied and surprising, but more error-prone., penaltiesrepetition penalty, n.: A setting that discourages an AI model from repeating itself by lowering the chances of words it has already used, so its text does not get stuck in loops., logit biaslogit bias, n.: A setting that lets developers make specific words more or less likely in an AI model's output — even ban a word outright — by adjusting the model's scores before it chooses. or grammar masks (constrained decodingconstrained decoding, n.: Forcing an AI model's output to follow a required format — such as valid code or a fill-in form — by blocking, at every step, any word that would break the rules.) — and selects a tokentoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens., which is appended and fed back, until an end-of-sequence token, a stop sequencestop sequence, n.: A piece of text that tells an AI model to stop writing as soon as it produces it — for example, stopping at a blank line or just before the next question. or a length limit is reached. Strategies divide into deterministic searches for high-probability outputs (greedy decodinggreedy decoding, n.: The simplest way for a language model to write: always pick the single most likely next word. It is fast and predictable but can produce dull or repetitive text., beam searchbeam search, n.: A search method that keeps several of the best partial answers at each step, rather than only the single best, to find a better overall result without trying every possibility.) and stochastic sampling, usually truncated to plausible candidates (top-ktop-k sampling, n.: A way of making AI-written text varied but sensible: at each step the model picks at random, but only among its k most likely next words, such as the top 40., top-ptop-p sampling, n.: A way of making AI-written text varied but sensible: the model picks at random among the smallest group of likely next words whose chances add up to a set share, such as 90%., min-pmin-p sampling, n.: A sampling method that lets the model consider any next word at least a set fraction as likely as its top choice, so the shortlist shrinks when the model is sure and grows when it is not.). The choice strongly affects quality: maximizing likelihood tends to give bland, repetitive text in open-ended generation, while untruncated sampling drifts into incoherence. Other methods spend extra computation at decoding time, such as speculative decodingspeculative decoding, n.: A speed-up trick in which a small, fast model guesses the next few words and the big model checks all the guesses at once, keeping the right ones: faster answers, same quality. for speed and best-of-n samplingbest-of-n sampling, n.: Asking an AI the same question several times and keeping whichever answer a scoring model or checker rates highest — a simple way to trade extra computing for better answers. or contrastive decoding for quality.
Same model, same prompt — switching the decoding from greedy to nucleus sampling fixed the looping.
- 2
Inference systems
In LLM serving, the token-by-token generation stage that follows processing of the prompt, typically limited by memory bandwidth rather than arithmetic; see decode phasedecode phase, n.: The second stage of an AI's reply, in which it writes the answer one word-piece at a time, each step depending on everything written so far. and prefillprefill, n.: The first stage of an AI's reply, in which it reads your whole prompt in one go before it starts writing. Long prompts make this stage take longer..
- 3
Other fields
In coding theory, recovering a transmitted message from a noisy received signal, as the Viterbi algorithm does for convolutional codes; in neuroscience, inferring stimuli, intentions or speech from recorded brain activity ('neural decoding'), increasingly with AI models. The language-model sense descends from the encoder–decoder vocabulary of machine translation.
Origin
From information theory, via statistical speech recognition, which treated finding the most probable word sequence as decoding a noisy channel (for example Lalit Bahl and Frederick Jelinek, 1975), and statistical machine translation, whose search programs were called decoders. Encoder–decoder neural translation (2014), with its 'decoder' network, carried the word into neural generation; Ari Holtzman and colleagues' 2019 study of 'neural text degeneration' made decoding strategy a research topic in its own right.
Related
- greedy decodinggreedy decoding, n.: The simplest way for a language model to write: always pick the single most likely next word. It is fast and predictable but can produce dull or repetitive text.
- beam searchbeam search, n.: A search method that keeps several of the best partial answers at each step, rather than only the single best, to find a better overall result without trying every possibility.
- temperaturetemperature, n.: A setting that controls how adventurous an AI model's word choices are. Low temperature makes it pick the safest, most likely words; high temperature makes it more varied and surprising, but more error-prone.
- top-p samplingtop-p sampling, n.: A way of making AI-written text varied but sensible: the model picks at random among the smallest group of likely next words whose chances add up to a set share, such as 90%.
- constrained decodingconstrained decoding, n.: Forcing an AI model's output to follow a required format — such as valid code or a fill-in form — by blocking, at every step, any word that would break the rules.
- speculative decodingspeculative decoding, n.: A speed-up trick in which a small, fast model guesses the next few words and the big model checks all the guesses at once, keeping the right ones: faster answers, same quality.
- stop sequencestop sequence, n.: A piece of text that tells an AI model to stop writing as soon as it produces it — for example, stopping at a blank line or just before the next question.
- decode phasedecode phase, n.: The second stage of an AI's reply, in which it writes the answer one word-piece at a time, each step depending on everything written so far.
Sources
- Holtzman, A., Buys, J., Du, L., Forbes, M. & Choi, Y. (2020). The Curious Case of Neural Text Degeneration. ICLR 2020. arxiv.org/abs/1904.09751
Mentioned in
AI watermarkingAI watermarking, n.: Hiding an invisible signal in AI-generated text, images, audio or video so that software can later tell the content was made by AI — like a banknote's watermark, but undetectable to the eye., language modellanguage model, n.: A system that has learned how likely different sequences of words are, so it can predict the next word in a text and, by doing that repeatedly, write text of its own., next-token predictionnext-token prediction, n.: The simple task most AI chatbots were trained on: given some text, guess the next small piece of it. Repeating that guess, piece by piece, produces whole answers., temperaturetemperature, n.: A setting that controls how adventurous an AI model's word choices are. Low temperature makes it pick the safest, most likely words; high temperature makes it more varied and surprising, but more error-prone., beam searchbeam search, n.: A search method that keeps several of the best partial answers at each step, rather than only the single best, to find a better overall result without trying every possibility., completioncompletion, n.: The text an AI language model writes in response to a prompt — literally, its continuation of the text it was given., constrained decodingconstrained decoding, n.: Forcing an AI model's output to follow a required format — such as valid code or a fill-in form — by blocking, at every step, any word that would break the rules., decode phasedecode phase, n.: The second stage of an AI's reply, in which it writes the answer one word-piece at a time, each step depending on everything written so far., greedy decodinggreedy decoding, n.: The simplest way for a language model to write: always pick the single most likely next word. It is fast and predictable but can produce dull or repetitive text., natural language generationnatural language generation, n.: Getting computers to produce human language — from turning a spreadsheet into a weather report to writing an essay or a chatbot's reply., repetition penaltyrepetition penalty, n.: A setting that discourages an AI model from repeating itself by lowering the chances of words it has already used, so its text does not get stuck in loops., samplingsampling, n.: Picking some items at random from a larger set or from a range of possibilities — surveying 1,000 voters, say, or letting an AI choose its next word by a weighted lottery., self-consistencyself-consistency, n.: Asking an AI the same question many times, letting it reason differently each time, then going with the answer it reaches most often — a vote among its own attempts., stop sequencestop sequence, n.: A piece of text that tells an AI model to stop writing as soon as it produces it — for example, stopping at a blank line or just before the next question., top-k samplingtop-k sampling, n.: A way of making AI-written text varied but sensible: at each step the model picks at random, but only among its k most likely next words, such as the top 40., top-p samplingtop-p sampling, n.: A way of making AI-written text varied but sensible: the model picks at random among the smallest group of likely next words whose chances add up to a set share, such as 90%., Viterbi algorithmViterbi algorithm, n.: A method that finds the likeliest hidden sequence behind some observations — such as each word's grammatical role in a sentence — by keeping only the best path to each point instead of trying every combination., diffusion language modeldiffusion language model, n.: A language model that writes by starting from a fully blanked-out answer and filling it in over several rounds, settling many words at once instead of writing strictly left to right., exposure biasexposure bias, n.: A weakness of models trained only on perfect examples: when they write on their own, one early mistake puts them in unfamiliar territory they never practiced, and errors can snowball., logit biaslogit bias, n.: A setting that lets developers make specific words more or less likely in an AI model's output — even ban a word outright — by adjusting the model's scores before it chooses., min-p samplingmin-p sampling, n.: A sampling method that lets the model consider any next word at least a set fraction as likely as its top choice, so the shortlist shrinks when the model is sure and grows when it is not., multi-query attentionmulti-query attention, n.: A cheaper form of attention in which all the attention heads share one store of information about earlier words instead of each keeping its own, making text generation faster and lighter on memory., multi-token predictionmulti-token prediction, n.: Training a language model to guess several upcoming words at once rather than only the next one, which can help it learn better and later write faster.