diffusion transformer
In plain wordsA diffusion image or video generator whose core network is a transformer — the architecture behind chatbots — instead of the older image-specific design, which lets it improve more predictably as it grows.
A diffusion modeldiffusion model, n.: An AI that learns to create images, sound or video by practicing noise removal: shown pictures buried under static, it learns to clean them up, then builds new ones starting from pure static. or flow matchingflow matching, n.: A way to train generators that learns, for every point between random noise and real data, which direction to move, so that noise can be steered smoothly into new examples. model whose denoising network is a transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it. reading the noisy latent as a sequence of patches, as a language model reads words, in place of a convolutional U-NetU-Net, prop. n.: An image network shaped like the letter U: one half shrinks the picture to understand it, the other enlarges it back, with shortcuts between the halves so fine detail is not lost.. The autoencoder latent is cut into patches (2 × 2 in the best original DiT), linearly embedded with positional information, processed by transformer blocks and projected back to a noise or velocity prediction; in the original design the timestep and class label condition each block through adaptive layer normalization, with every block initialized to the identity function ('adaLN-Zero'). DiT quality improved steadily with forward-pass compute, from larger models or smaller patches, as in language models. Text-to-image and video variants add text either through cross-attentioncross-attention, n.: Attention between two different inputs: one sequence consults another, as when a translation model writing French looks back at the English sentence, or an image generator consults the text prompt. or by joint attention over text and image tokens, as in the MMDiT of Stable Diffusion 3, which keeps separate weights for each modality; for video, patches span both space and time. DiT-style backbones were used in SoraSora, prop. n.: OpenAI's video-generating AI, unveiled in 2024 with startlingly realistic clips; a 2025 version added sound and a social app, but OpenAI shut the service down in 2026. and underlie FLUXFLUX, prop. n.: A family of image-generating AI models from the German start-up Black Forest Labs, founded by researchers behind Stable Diffusion; some versions can be downloaded and run freely., Stable Diffusion 3 and most image and video generators of 2024–2026.
Origin
Introduced by William Peebles (UC Berkeley) and Saining Xie (New York University) in 'Scalable Diffusion Models with Transformers' (posted December 2022; ICCV 2023), whose DiT-XL/2 reached an FID of 2.27 on class-conditional ImageNet at 256 × 256. Peebles went on to co-lead OpenAI's Sora.
Related
- diffusion modeldiffusion model, n.: An AI that learns to create images, sound or video by practicing noise removal: shown pictures buried under static, it learns to clean them up, then builds new ones starting from pure static.
- transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it.
- latent diffusion modellatent diffusion model, n.: A diffusion image generator that works on a shrunken, compressed version of each picture instead of the full-size pixels, which makes it far cheaper to train and run.
- U-NetU-Net, prop. n.: An image network shaped like the letter U: one half shrinks the picture to understand it, the other enlarges it back, with shortcuts between the halves so fine detail is not lost.
- SoraSora, prop. n.: OpenAI's video-generating AI, unveiled in 2024 with startlingly realistic clips; a 2025 version added sound and a social app, but OpenAI shut the service down in 2026.
- vision transformervision transformer, n.: A way of applying the transformer, the design behind chatbots, to pictures: the image is cut into small square tiles that the model reads like words in a sentence.
- patch embeddingpatch embedding, n.: The first step of a vision transformer: cutting an image into small square tiles and turning each tile into a list of numbers that the model can process like a word.
Sources
- Peebles, W. & Xie, S. (2023). Scalable Diffusion Models with Transformers. ICCV 2023. arXiv:2212.09748. arxiv.org/abs/2212.09748
- Esser, P. et al. (2024). Scaling Rectified Flow Transformers for High-Resolution Image Synthesis. arXiv:2403.03206 (ICML 2024). arxiv.org/abs/2403.03206
Mentioned in
diffusion modeldiffusion model, n.: An AI that learns to create images, sound or video by practicing noise removal: shown pictures buried under static, it learns to clean them up, then builds new ones starting from pure static., Stable DiffusionStable Diffusion, prop. n.: A widely used image-generating AI whose model files were released for anyone to download, so people could run it on their own computers and adapt it., text-to-image modeltext-to-image model, n.: An AI system that creates a picture from a written description, such as 'a fox reading a newspaper, in watercolor', usually within seconds., text-to-video modeltext-to-video model, n.: An AI that creates a short video clip — increasingly with matching sound — from a written description, such as 'a paper boat sailing down a rainy street'., transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it., flow matchingflow matching, n.: A way to train generators that learns, for every point between random noise and real data, which direction to move, so that noise can be steered smoothly into new examples., FLUXFLUX, prop. n.: A family of image-generating AI models from the German start-up Black Forest Labs, founded by researchers behind Stable Diffusion; some versions can be downloaded and run freely., latent diffusion modellatent diffusion model, n.: A diffusion image generator that works on a shrunken, compressed version of each picture instead of the full-size pixels, which makes it far cheaper to train and run., SoraSora, prop. n.: OpenAI's video-generating AI, unveiled in 2024 with startlingly realistic clips; a 2025 version added sound and a social app, but OpenAI shut the service down in 2026., U-NetU-Net, prop. n.: An image network shaped like the letter U: one half shrinks the picture to understand it, the other enlarges it back, with shortcuts between the halves so fine detail is not lost., rectified flowrectified flow, n.: A generator trained to move random noise toward real data along straight lines, which can then be straightened further so that a picture comes out in one or a few steps., world foundation modelworld foundation model, n.: A large AI model trained on vast amounts of video that can imagine how a scene will unfold, used to create realistic practice worlds for robots and self-driving cars.