classifier-free guidance
In plain wordsA setting that makes an image generator follow the prompt more strictly by comparing what it would draw with and without the prompt, then exaggerating the difference.
A sampling technique that strengthens the conditioning of a diffusion modeldiffusion model, n.: An AI that learns to create images, sound or video by practicing noise removal: shown pictures buried under static, it learns to clean them up, then builds new ones starting from pure static. or flow matchingflow matching, n.: A way to train generators that learns, for every point between random noise and real data, which direction to move, so that noise can be steered smoothly into new examples. model without any external classifier. One network is trained both with its condition c (such as a text prompt) and, for a random fraction of examples (often 10–20 percent), with the condition replaced by a null input, so that it learns conditional and unconditional predictions. At each sampling step both are computed and extrapolated: the output is the unconditional prediction plus w times the difference between the conditional and unconditional predictions. A guidance scale w of 1 gives plain conditional sampling; values around 3–10 typically improve prompt adherence and image quality but reduce diversity and, when too high, oversaturate colors. Because it doubles the network evaluations per step, guidance is often distilled into the model. Replacing the unconditional input with an undesired prompt yields the negative promptnegative prompt, n.: A second prompt listing what you do not want in a generated image — 'blurry, extra fingers, watermark' — which the generator then steers away from..
At a CFG scale of 12 the colors burn out; try 6.
Origin
Proposed by Jonathan Ho and Tim Salimans of Google in a short paper at a NeurIPS 2021 workshop, posted to arXiv in July 2022. OpenAI's GLIDE (December 2021) found that human raters preferred it to CLIP guidance; it was adopted by DALL·E 2, Imagen and Stable Diffusion and became a default in nearly every text-to-image system.
Usage note
Conventions differ: some papers parameterize the weight so that 0, not 1, means no guidance. 'Guidance' here is a sampling technique and has nothing to do with safety guardrailsguardrails, n.: Safety checks built around an AI system — filters on what goes in and comes out, and limits on what it may do — to keep it from producing harmful or off-topic content or leaking data..
Related
- classifier guidanceclassifier guidance, n.: A way to push a diffusion image generator toward a requested category by consulting a separate image-recognition model at every step, trading variety for images that more clearly show the target.
- negative promptnegative prompt, n.: A second prompt listing what you do not want in a generated image — 'blurry, extra fingers, watermark' — which the generator then steers away from.
- diffusion modeldiffusion model, n.: An AI that learns to create images, sound or video by practicing noise removal: shown pictures buried under static, it learns to clean them up, then builds new ones starting from pure static.
- text-to-image modeltext-to-image model, n.: An AI system that creates a picture from a written description, such as 'a fox reading a newspaper, in watercolor', usually within seconds.
- diffusion samplerdiffusion sampler, n.: The step-by-step routine a diffusion model follows to turn noise into a finished picture; the choice of sampler and number of steps trades speed against quality.
- temperaturetemperature, n.: A setting that controls how adventurous an AI model's word choices are. Low temperature makes it pick the safest, most likely words; high temperature makes it more varied and surprising, but more error-prone.
- classifier guidanceclassifier guidance, n.: A way to push a diffusion image generator toward a requested category by consulting a separate image-recognition model at every step, trading variety for images that more clearly show the target.
Sources
- Ho, J. & Salimans, T. (2022). Classifier-Free Diffusion Guidance. arXiv:2207.12598 (NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications). arxiv.org/abs/2207.12598
- Nichol, A. et al. (2021). GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models. arXiv:2112.10741 (ICML 2022). arxiv.org/abs/2112.10741
Mentioned in
diffusion modeldiffusion model, n.: An AI that learns to create images, sound or video by practicing noise removal: shown pictures buried under static, it learns to clean them up, then builds new ones starting from pure static., text-to-image modeltext-to-image model, n.: An AI system that creates a picture from a written description, such as 'a fox reading a newspaper, in watercolor', usually within seconds., flow matchingflow matching, n.: A way to train generators that learns, for every point between random noise and real data, which direction to move, so that noise can be steered smoothly into new examples., negative promptnegative prompt, n.: A second prompt listing what you do not want in a generated image — 'blurry, extra fingers, watermark' — which the generator then steers away from., steeringsteering, n.: Guiding what an AI model does — its topic, tone or behavior — while it runs, by changing its instructions, its internal signals or how its next word is chosen, rather than by retraining it., classifier guidanceclassifier guidance, n.: A way to push a diffusion image generator toward a requested category by consulting a separate image-recognition model at every step, trading variety for images that more clearly show the target., CLIP scoreCLIP score, n.: A score for how well a picture matches a text description, computed by an AI model that was trained to pair images with their captions., diffusion distillationdiffusion distillation, n.: Training a fast image generator to imitate a slow diffusion model, so that a picture that once took fifty steps can be made in one to four., plug and play language modelplug and play language model, n.: An early way to steer what a language model writes — its topic or mood — by letting a small add-on classifier nudge the model's internal state at each word, without retraining the model., score distillation samplingscore distillation sampling, n.: A trick for creating 3D objects from a text description by repeatedly asking a 2D image generator whether pictures of the object, seen from many angles, look right — and adjusting the object until they do.