Skip to content
Last Dictionary
Plane
Language Models & TransformersGenerative Media
emerging

diffusion language model

nounAlso: DLM, dLLM, diffusion LLM, masked diffusion language model, text diffusion

In plain wordsA language model that writes by starting from a fully blanked-out answer and filling it in over several rounds, settling many words at once instead of writing strictly left to right.

  1. A language modellanguage model, n.: A system that has learned how likely different sequences of words are, so it can predict the next word in a text and, by doing that repeatedly, write text of its own. that generates text by iteratively denoising a whole sequence rather than producing it one token at a time. Most current designs use masked (absorbing-state) discrete diffusion: in training, a random fraction of tokens is replaced by a mask symbol and a bidirectional transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it. learns to predict them, which yields a bound on likelihood; at inference, generation starts from an all-mask sequence of chosen length and unmasks tokens over a number of steps, typically committing the most confident predictions first. Because many tokens are decided in parallel, the number of steps trades quality for speed, and infilling, editing and use of context on both sides come naturally. Weaknesses include fixed output lengths, no straightforward KV cachingKV cache, n.: A memory store where a chatbot keeps its working notes on everything already read and written, so each new word does not require re-reading the whole conversation from scratch., and quality that has generally trailed comparable autoregressive modelsautoregressive model, n.: A model that produces a sequence one piece at a time, each new piece depending on the pieces already produced — like writing a sentence word by word without going back., although LLaDA 8B (2025) was reported competitive with LLaMA 3 8B. Inception Labs' commercial Mercury models (from February 2025; Mercury 2, with reasoning, February 2026) are marketed chiefly on speed; Google's Gemini Diffusion, announced in May 2025, remained an experimental demo as of October 2026.

    The diffusion language model fills a 256-token answer in 32 denoising steps.

Origin

Discrete diffusion for text builds on D3PM (Jacob Austin and colleagues, 2021); Diffusion-LM (Xiang Lisa Li and colleagues, 2022) diffused continuous word embeddings. Simplified masked-diffusion objectives followed in 2024, and LLaDA (Shen Nie and colleagues, February 2025) scaled the approach to 8 billion parameters trained from scratch.

Related

See also
  • diffusion modeldiffusion model, n.: An AI that learns to create images, sound or video by practicing noise removal: shown pictures buried under static, it learns to clean them up, then builds new ones starting from pure static.
  • masked language modelmasked language model, n.: A model trained by hiding some words in a sentence and learning to guess them from the words on both sides, like a fill-in-the-blanks exercise. BERT is the famous example.
  • autoregressive modelautoregressive model, n.: A model that produces a sequence one piece at a time, each new piece depending on the pieces already produced — like writing a sentence word by word without going back.
  • next-token predictionnext-token prediction, n.: The simple task most AI chatbots were trained on: given some text, guess the next small piece of it. Repeating that guess, piece by piece, produces whole answers.
  • decodingdecoding, n.: The method a language model uses to turn its predictions into actual text — for example, always picking the most likely next word, or choosing somewhat at random among the likely ones.
  • multi-token predictionmulti-token prediction, n.: Training a language model to guess several upcoming words at once rather than only the next one, which can help it learn better and later write faster.
Contrast with
  • autoregressive modelautoregressive model, n.: A model that produces a sequence one piece at a time, each new piece depending on the pieces already produced — like writing a sentence word by word without going back.

Sources

  1. Austin, J., Johnson, D. D., Ho, J., Tarlow, D. & van den Berg, R. (2021). Structured Denoising Diffusion Models in Discrete State-Spaces. NeurIPS 2021.
  2. Nie, S., Zhu, F., You, Z. et al. (2025). Large Language Diffusion Models. arXiv:2502.09992. arxiv.org/abs/2502.09992

Mentioned in

diffusion modeldiffusion model, n.: An AI that learns to create images, sound or video by practicing noise removal: shown pictures buried under static, it learns to clean them up, then builds new ones starting from pure static., language modellanguage model, n.: A system that has learned how likely different sequences of words are, so it can predict the next word in a text and, by doing that repeatedly, write text of its own., next-token predictionnext-token prediction, n.: The simple task most AI chatbots were trained on: given some text, guess the next small piece of it. Repeating that guess, piece by piece, produces whole answers., autoregressive modelautoregressive model, n.: A model that produces a sequence one piece at a time, each new piece depending on the pieces already produced — like writing a sentence word by word without going back., masked language modelmasked language model, n.: A model trained by hiding some words in a sentence and learning to guess them from the words on both sides, like a fill-in-the-blanks exercise. BERT is the famous example.