multi-token prediction
In plain wordsTraining a language model to guess several upcoming words at once rather than only the next one, which can help it learn better and later write faster.
A training objective in which a language modellanguage model, n.: A system that has learned how likely different sequences of words are, so it can predict the next word in a text and, by doing that repeatedly, write text of its own. predicts several future tokenstoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens. at each position instead of only the next. In the formulation of Fabian Gloeckle and colleagues (2024), a shared transformer trunk feeds n independent output heads, the i-th predicting the token i steps ahead, and their losses are summed; 13-billion-parameter code models trained this way solved 12% more HumanEval and 17% more MBPP problems than next-token baselines, and 4-token models decoded up to three times faster by drafting their own continuations. DeepSeek-V3 (December 2024) used sequential MTP modules that keep the causal chain between predicted tokens, to densify the training signal and to supply draft tokens for speculative decodingspeculative decoding, n.: A speed-up trick in which a small, fast model guesses the next few words and the big model checks all the guesses at once, keeping the right ones: faster answers, same quality.. Gains appear mainly in larger models and on structured outputs such as code; the extra heads can be dropped at inference.
With MTP heads as drafters, we get about 1.8 accepted tokens per forward pass.
Origin
Predicting several future tokens was explored in blockwise parallel decoding (Mitchell Stern, Noam Shazeer and Jakob Uszkoreit, 2018) and ProphetNet (Weizhen Qi and colleagues, 2020). The term was popularized by Gloeckle and colleagues at Meta (April 2024) and by its use in DeepSeek-V3.
Related
- next-token predictionnext-token prediction, n.: The simple task most AI chatbots were trained on: given some text, guess the next small piece of it. Repeating that guess, piece by piece, produces whole answers.
- speculative decodingspeculative decoding, n.: A speed-up trick in which a small, fast model guesses the next few words and the big model checks all the guesses at once, keeping the right ones: faster answers, same quality.
- decodingdecoding, n.: The method a language model uses to turn its predictions into actual text — for example, always picking the most likely next word, or choosing somewhat at random among the likely ones.
- autoregressive modelautoregressive model, n.: A model that produces a sequence one piece at a time, each new piece depending on the pieces already produced — like writing a sentence word by word without going back.
- pretrainingpretraining, n.: The first and usually the most expensive stage of building a modern AI model, in which it learns general patterns from enormous amounts of raw text, images or code before being specialized for particular uses.
- next-token predictionnext-token prediction, n.: The simple task most AI chatbots were trained on: given some text, guess the next small piece of it. Repeating that guess, piece by piece, produces whole answers.
Sources
- Gloeckle, F., Youbi Idrissi, B., Rozière, B., Lopez-Paz, D. & Synnaeve, G. (2024). Better & Faster Large Language Models via Multi-token Prediction. ICML 2024. arxiv.org/abs/2404.19737
- DeepSeek-AI (2024). DeepSeek-V3 Technical Report. arXiv:2412.19437. arxiv.org/abs/2412.19437
Mentioned in
next-token predictionnext-token prediction, n.: The simple task most AI chatbots were trained on: given some text, guess the next small piece of it. Repeating that guess, piece by piece, produces whole answers., autoregressive modelautoregressive model, n.: A model that produces a sequence one piece at a time, each new piece depending on the pieces already produced — like writing a sentence word by word without going back., decode phasedecode phase, n.: The second stage of an AI's reply, in which it writes the answer one word-piece at a time, each step depending on everything written so far., speculative decodingspeculative decoding, n.: A speed-up trick in which a small, fast model guesses the next few words and the big model checks all the guesses at once, keeping the right ones: faster answers, same quality., diffusion language modeldiffusion language model, n.: A language model that writes by starting from a fully blanked-out answer and filling it in over several rounds, settling many words at once instead of writing strictly left to right., Jakob UszkoreitJakob Uszkoreit, prop. n.: A German computer scientist, one of the eight authors of the transformer paper, credited with first proposing to build language models on attention alone.