latent diffusion model
In plain wordsA diffusion image generator that works on a shrunken, compressed version of each picture instead of the full-size pixels, which makes it far cheaper to train and run.
A diffusion modeldiffusion model, n.: An AI that learns to create images, sound or video by practicing noise removal: shown pictures buried under static, it learns to clean them up, then builds new ones starting from pure static. that generates compressed codes instead of pixels: a separately trained autoencoderautoencoder, n.: A neural network trained to squeeze its input into a compact code and then rebuild the original from that code. Learning to compress this way forces it to capture what matters in the data. learns to squeeze images into a small latent tensor and back, and the diffusion process runs entirely on those latents. The encoder — typically a VAEvariational autoencoder, n.: A neural network that learns to squeeze data into a compact, slightly fuzzy code and rebuild it, arranged so that decoding random codes produces new, plausible examples. trained with perceptualperceptual loss, n.: A way of judging how different two images look to a person by comparing what a trained image-recognition network 'sees' in each, instead of comparing pixels one by one. and adversarial losses — turns a 512 × 512 image into, for example, a 64 × 64 × 4 latent (8 times smaller per side), and its decoder renders generated latents back into pixels. Text and other conditions usually reach the denoising network through cross-attentioncross-attention, n.: Attention between two different inputs: one sequence consults another, as when a translation model writing French looks back at the English sentence, or an image generator consults the text prompt. layers. Because the autoencoder has already stripped out fine detail the eye barely registers, the generative model can spend its capacity on content and composition, which cuts training and sampling compute by large factors while preserving quality. Most modern image and video generators, including Stable DiffusionStable Diffusion, prop. n.: A widely used image-generating AI whose model files were released for anyone to download, so people could run it on their own computers and adapt it. and FLUXFLUX, prop. n.: A family of image-generating AI models from the German start-up Black Forest Labs, founded by researchers behind Stable Diffusion; some versions can be downloaded and run freely., are latent diffusion or latent flow matchingflow matching, n.: A way to train generators that learns, for every point between random noise and real data, which direction to move, so that noise can be steered smoothly into new examples. models, as was SoraSora, prop. n.: OpenAI's video-generating AI, unveiled in 2024 with startlingly realistic clips; a 2025 version added sound and a social app, but OpenAI shut the service down in 2026., with temporal compression added for video.
Training in latent space let us fit the whole model on eight GPUs.
Origin
Introduced by Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser and Björn Ommer (LMU Munich, Heidelberg University and Runway) in 'High-Resolution Image Synthesis with Latent Diffusion Models' (posted December 2021; CVPR 2022). Their open text-to-image model, trained with Stability AI compute, became Stable Diffusion in August 2022.
Related
- diffusion modeldiffusion model, n.: An AI that learns to create images, sound or video by practicing noise removal: shown pictures buried under static, it learns to clean them up, then builds new ones starting from pure static.
- Stable DiffusionStable Diffusion, prop. n.: A widely used image-generating AI whose model files were released for anyone to download, so people could run it on their own computers and adapt it.
- variational autoencodervariational autoencoder, n.: A neural network that learns to squeeze data into a compact, slightly fuzzy code and rebuild it, arranged so that decoding random codes produces new, plausible examples.
- VQGANVQGAN, n.: An image compressor that turns a picture into a small grid of codebook 'tiles' while keeping it sharp, used as the vocabulary of many image generators and of an early wave of AI art.
- latent spacelatent space, n.: The hidden internal map a model builds of its data, where each item becomes a point and similar items sit near each other. Moving smoothly through it gradually changes what the model represents or generates.
- cross-attentioncross-attention, n.: Attention between two different inputs: one sequence consults another, as when a translation model writing French looks back at the English sentence, or an image generator consults the text prompt.
- diffusion transformerdiffusion transformer, n.: A diffusion image or video generator whose core network is a transformer — the architecture behind chatbots — instead of the older image-specific design, which lets it improve more predictably as it grows.
Sources
- Rombach, R., Blattmann, A., Lorenz, D., Esser, P. & Ommer, B. (2022). High-Resolution Image Synthesis with Latent Diffusion Models. CVPR 2022. arXiv:2112.10752. arxiv.org/abs/2112.10752
Mentioned in
diffusion modeldiffusion model, n.: An AI that learns to create images, sound or video by practicing noise removal: shown pictures buried under static, it learns to clean them up, then builds new ones starting from pure static., Stable DiffusionStable Diffusion, prop. n.: A widely used image-generating AI whose model files were released for anyone to download, so people could run it on their own computers and adapt it., text-to-image modeltext-to-image model, n.: An AI system that creates a picture from a written description, such as 'a fox reading a newspaper, in watercolor', usually within seconds., text-to-video modeltext-to-video model, n.: An AI that creates a short video clip — increasingly with matching sound — from a written description, such as 'a paper boat sailing down a rainy street'., autoencoderautoencoder, n.: A neural network trained to squeeze its input into a compact code and then rebuild the original from that code. Learning to compress this way forces it to capture what matters in the data., cross-attentioncross-attention, n.: Attention between two different inputs: one sequence consults another, as when a translation model writing French looks back at the English sentence, or an image generator consults the text prompt., CVPRCVPR, prop. n.: The leading yearly conference on computer vision — teaching computers to understand images and video — and one of the most cited publication venues in any field of science., diffusion transformerdiffusion transformer, n.: A diffusion image or video generator whose core network is a transformer — the architecture behind chatbots — instead of the older image-specific design, which lets it improve more predictably as it grows., encoder–decoder architectureencoder–decoder architecture, n.: A two-part model design: one part, the encoder, reads the input and condenses it into an internal summary; the other, the decoder, uses that summary to produce the output., FLUXFLUX, prop. n.: A family of image-generating AI models from the German start-up Black Forest Labs, founded by researchers behind Stable Diffusion; some versions can be downloaded and run freely., latent spacelatent space, n.: The hidden internal map a model builds of its data, where each item becomes a point and similar items sit near each other. Moving smoothly through it gradually changes what the model represents or generates., Stability AIStability AI, prop. n.: A London company that paid for and released Stable Diffusion, a free image-generating AI anyone could download, and later models for video, audio and 3D., variational autoencodervariational autoencoder, n.: A neural network that learns to squeeze data into a compact, slightly fuzzy code and rebuild it, arranged so that decoding random codes produces new, plausible examples., Black Forest LabsBlack Forest Labs, prop. n.: A German AI company founded in 2024 by original creators of Stable Diffusion, which makes the FLUX image-generation models., image tokenizerimage tokenizer, n.: A component that chops a picture into a short sequence of 'visual words' that a language-style AI model can read or write, and turns such sequences back into pictures., perceptual lossperceptual loss, n.: A way of judging how different two images look to a person by comparing what a trained image-recognition network 'sees' in each, instead of comparing pixels one by one., RunwayRunway, prop. n.: A New York company that makes AI tools for generating and editing video, used by filmmakers and artists; it helped create the image generator Stable Diffusion., VQGANVQGAN, n.: An image compressor that turns a picture into a small grid of codebook 'tiles' while keeping it sharp, used as the vocabulary of many image generators and of an early wave of AI art.