Skip to content
Last Dictionary
Plane
Language Models & TransformersGenerative MediaMathematics & Statistics

autoregressive model

nounAlso: autoregressive, AR model, autoregression, autoregressive generation

In plain wordsA model that produces a sequence one piece at a time, each new piece depending on the pieces already produced — like writing a sentence word by word without going back.

  1. 1

    Machine learning

    A generative modelgenerative model, n.: A model that learns what a kind of data looks like — faces, sentences, songs — well enough to produce new examples that resemble the originals without copying them. that factorizes the joint distribution of a sequence into a product of conditional distributions, each element predicted from those before it, and generates by sampling one element at a time and feeding it back as input. GPTGPT, abbr.: Short for 'generative pre-trained transformer', the family of OpenAI models behind ChatGPT. The letters are also used loosely for any similar chatbot model.-style decoder-only transformersdecoder-only transformer, n.: The kind of transformer behind most chatbots: a single stack that reads text left to right and continues it, without a separate part that first encodes an input. are autoregressive over tokenstoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens.; PixelCNN and WaveNet applied the idea to pixels and audio samples, and autoregressive image generationautoregressive image generation, n.: Making a picture one piece at a time, each piece predicted from those already made — the way chatbots write text word by word. applies it to image tokens. Training is efficient because all conditionals are computed in parallel under a causal maskcausal mask, n.: A rule inside text-generating AI that stops each word from seeing the words after it, so the model learns to predict what comes next without being able to cheat., but generation is sequential, so latency grows with output length — the bottleneck that speculative decodingspeculative decoding, n.: A speed-up trick in which a small, fast model guesses the next few words and the big model checks all the guesses at once, keeping the right ones: faster answers, same quality., multi-token predictionmulti-token prediction, n.: Training a language model to guess several upcoming words at once rather than only the next one, which can help it learn better and later write faster. and diffusion language modelsdiffusion language model, n.: A language model that writes by starting from a fully blanked-out answer and filling it in over several rounds, settling many words at once instead of writing strictly left to right. attack. Because the model conditions on its own earlier outputs, early mistakes can compound (exposure biasexposure bias, n.: A weakness of models trained only on perfect examples: when they write on their own, one early mistake puts them in unfamiliar territory they never practiced, and errors can snowball.).

    Autoregressive decoding means a 2,000-token answer needs 2,000 sequential forward passes.

  2. 2

    Statistics

    A time-series model in which a variable is regressed on its own previous values, as in the AR(p) model where x_t is a linear combination of the p preceding values plus noise; a building block of ARIMAARIMA, n.: A classic statistical recipe for forecasting a series of numbers over time: remove any steady trend, then predict the next value from recent values and recent prediction errors. forecasting. The machine-learning sense generalizes this idea to arbitrary learned conditional distributions.

p(x1:T)=∏t=1Tp(xt∣x<t)p(x_{1:T}) = \prod_{t=1}^{T} p(x_t \mid x_{<t})

Origin

Autoregressive models in statistics date to G. Udny Yule's 1927 analysis of sunspot numbers and Gilbert Walker's work of 1931. In machine learning, neural autoregressive density estimators (Hugo Larochelle and Iain Murray, 2011) and WaveNet and PixelCNN (2016) preceded autoregressive transformers such as GPT (2018).

Related

See also
  • next-token predictionnext-token prediction, n.: The simple task most AI chatbots were trained on: given some text, guess the next small piece of it. Repeating that guess, piece by piece, produces whole answers.
  • causal language modelcausal language model, n.: A language model that reads text only from left to right, predicting each word from the words before it. The models behind today's chatbots are of this kind.
  • decoder-only transformerdecoder-only transformer, n.: The kind of transformer behind most chatbots: a single stack that reads text left to right and continues it, without a separate part that first encodes an input.
  • exposure biasexposure bias, n.: A weakness of models trained only on perfect examples: when they write on their own, one early mistake puts them in unfamiliar territory they never practiced, and errors can snowball.
  • diffusion language modeldiffusion language model, n.: A language model that writes by starting from a fully blanked-out answer and filling it in over several rounds, settling many words at once instead of writing strictly left to right.
  • ARIMAARIMA, n.: A classic statistical recipe for forecasting a series of numbers over time: remove any steady trend, then predict the next value from recent values and recent prediction errors.
Contrast with
  • diffusion language modeldiffusion language model, n.: A language model that writes by starting from a fully blanked-out answer and filling it in over several rounds, settling many words at once instead of writing strictly left to right.
  • masked language modelmasked language model, n.: A model trained by hiding some words in a sentence and learning to guess them from the words on both sides, like a fill-in-the-blanks exercise. BERT is the famous example.

Mentioned in

diffusion modeldiffusion model, n.: An AI that learns to create images, sound or video by practicing noise removal: shown pictures buried under static, it learns to clean them up, then builds new ones starting from pure static., language modellanguage model, n.: A system that has learned how likely different sequences of words are, so it can predict the next word in a text and, by doing that repeatedly, write text of its own., next-token predictionnext-token prediction, n.: The simple task most AI chatbots were trained on: given some text, guess the next small piece of it. Repeating that guess, piece by piece, produces whole answers., ARIMAARIMA, n.: A classic statistical recipe for forecasting a series of numbers over time: remove any steady trend, then predict the next value from recent values and recent prediction errors., categorical distributioncategorical distribution, n.: The probabilities for a single pick among a fixed set of options, like a weighted die. A language model's guess about the next word is one of these, spread over its whole vocabulary., causal language modelcausal language model, n.: A language model that reads text only from left to right, predicting each word from the words before it. The models behind today's chatbots are of this kind., causal maskcausal mask, n.: A rule inside text-generating AI that stops each word from seeing the words after it, so the model learns to predict what comes next without being able to cheat., chain rulechain rule, n.: A rule for working out how a change ripples through a chain of steps by combining the effect at each step. It is how AI training traces each error back through every layer., completioncompletion, n.: The text an AI language model writes in response to a prompt — literally, its continuation of the text it was given., conditional probabilityconditional probability, n.: The chance of something once you already know that something else is true — for example, the chance of rain given that the sky is cloudy., continuous batchingcontinuous batching, n.: A way for an AI server to handle many conversations together, letting new ones join and finished ones leave after every word, instead of waiting for a whole group to finish., decode phasedecode phase, n.: The second stage of an AI's reply, in which it writes the answer one word-piece at a time, each step depending on everything written so far., density estimationdensity estimation, n.: Working out from a sample of data how likely different values are, by drawing a smooth curve or surface showing where data points bunch up and where they are rare., encoder–decoder architectureencoder–decoder architecture, n.: A two-part model design: one part, the encoder, reads the input and condenses it into an internal summary; the other, the decoder, uses that summary to produce the output., generative modelgenerative model, n.: A model that learns what a kind of data looks like — faces, sentences, songs — well enough to produce new examples that resemble the originals without copying them., maximum likelihood estimationmaximum likelihood estimation, n.: Tuning a model's settings until the real data it learns from looks as likely, and so as unsurprising, to the model as possible. Most AI training, including the pretraining of chatbots, is a version of this., prefillprefill, n.: The first stage of an AI's reply, in which it reads your whole prompt in one go before it starts writing. Long prompts make this stage take longer., sequence-to-sequence modelsequence-to-sequence model, n.: A model that turns one sequence into another, such as a French sentence into English or speech into text, by first reading the whole input and then writing the output step by step., speculative decodingspeculative decoding, n.: A speed-up trick in which a small, fast model guesses the next few words and the big model checks all the guesses at once, keeping the right ones: faster answers, same quality., teacher forcingteacher forcing, n.: Training a model that writes one word at a time by always feeding it the correct previous words, rather than its own possibly mistaken guesses, while it learns to predict the next one., WaveNetWaveNet, prop. n.: A DeepMind model that generates speech and other sounds one tiny audio sample at a time, producing voices far more natural than earlier computer speech., autoregressive image generationautoregressive image generation, n.: Making a picture one piece at a time, each piece predicted from those already made — the way chatbots write text word by word., diffusion language modeldiffusion language model, n.: A language model that writes by starting from a fully blanked-out answer and filling it in over several rounds, settling many words at once instead of writing strictly left to right., exposure biasexposure bias, n.: A weakness of models trained only on perfect examples: when they write on their own, one early mistake puts them in unfamiliar territory they never practiced, and errors can snowball.