Skip to content
Last Dictionary
Plane
Generative MediaDeep Learning

VQ-VAE

nounAlso: vector-quantized variational autoencoder, vector quantised VAE, VQ-VAE-2

In plain wordsA network that compresses an image or sound into a short list of entries from a learned 'codebook' — like describing a picture with a fixed set of tiles — which other models can then generate.

  1. An autoencoderautoencoder, n.: A neural network trained to squeeze its input into a compact code and then rebuild the original from that code. Learning to compress this way forces it to capture what matters in the data. with a discrete latent representation. The encoder outputs a grid (or sequence) of continuous vectors, each replaced by its nearest neighbor in a learned codebook of K embedding vectors; the decoder reconstructs the input from these quantized vectors. Because the nearest-neighbor lookup has no gradient, training copies the decoder's gradient straight through to the encoder (the straight-through estimator) and adds codebook and commitment losses that pull codebook entries and encoder outputs toward each other. The result turns images, audio or video into sequences of integer codes — tokenstoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens. — on which a separate autoregressive modelautoregressive model, n.: A model that produces a sequence one piece at a time, each new piece depending on the pieces already produced — like writing a sentence word by word without going back. (originally PixelCNN, later transformerstransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it.) is trained as a prior to generate new data. Discrete codes also avoid the posterior collapse that afflicts VAEsvariational autoencoder, n.: A neural network that learns to squeeze data into a compact, slightly fuzzy code and rebuild it, arranged so that decoding random codes produces new, plausible examples. with powerful decoders. VQ-VAE is the ancestor of VQGANVQGAN, n.: An image compressor that turns a picture into a small grid of codebook 'tiles' while keeping it sharp, used as the vocabulary of many image generators and of an early wave of AI art., of the discrete VAE in the first DALL·EDALL·E, prop. n.: OpenAI's pioneering series of image generators, named after the painter Salvador Dalí and the robot WALL·E, which showed in 2021–2023 that AI could draw almost anything described in words. and of many imageimage tokenizer, n.: A component that chops a picture into a short sequence of 'visual words' that a language-style AI model can read or write, and turns such sequences back into pictures. and audio tokenizersneural audio codec, n.: A neural network that squeezes sound into a very compact stream of numbers and rebuilds it, used both to send audio cheaply and to let AI models 'write' speech and music piece by piece..

Origin

Introduced by Aäron van den Oord, Oriol Vinyals and Koray Kavukcuoglu of DeepMind in 'Neural Discrete Representation Learning' (posted November 2017; NeurIPS 2017). VQ-VAE-2 (Ali Razavi, van den Oord and Vinyals, 2019) added a multiscale hierarchy, and OpenAI's Jukebox (2020) used a multiscale VQ-VAE for raw-audio music.

Usage note

Despite the name, the original VQ-VAE is trained much like a deterministic autoencoder; the 'variational' framing treats the quantized code as a one-hot posterior with a uniform prior.

Related

See also
  • VQGANVQGAN, n.: An image compressor that turns a picture into a small grid of codebook 'tiles' while keeping it sharp, used as the vocabulary of many image generators and of an early wave of AI art.
  • image tokenizerimage tokenizer, n.: A component that chops a picture into a short sequence of 'visual words' that a language-style AI model can read or write, and turns such sequences back into pictures.
  • variational autoencodervariational autoencoder, n.: A neural network that learns to squeeze data into a compact, slightly fuzzy code and rebuild it, arranged so that decoding random codes produces new, plausible examples.
  • neural audio codecneural audio codec, n.: A neural network that squeezes sound into a very compact stream of numbers and rebuilds it, used both to send audio cheaply and to let AI models 'write' speech and music piece by piece.
  • autoregressive image generationautoregressive image generation, n.: Making a picture one piece at a time, each piece predicted from those already made — the way chatbots write text word by word.
  • tokentoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens.

Sources

  1. van den Oord, A., Vinyals, O. & Kavukcuoglu, K. (2017). Neural Discrete Representation Learning. arXiv:1711.00937 (NeurIPS 2017). arxiv.org/abs/1711.00937

Mentioned in

DALL·EDALL·E, prop. n.: OpenAI's pioneering series of image generators, named after the painter Salvador Dalí and the robot WALL·E, which showed in 2021–2023 that AI could draw almost anything described in words., music generationmusic generation, n.: Using AI to compose or produce music — from melodies written as notes to complete songs with vocals — often from a short text description like 'upbeat jazz with saxophone'., quantizationquantization, n.: Storing an AI model's numbers with fewer digits, like rounding prices to the nearest dollar, so the model takes less memory and runs faster, usually with only a small loss of accuracy., variational autoencodervariational autoencoder, n.: A neural network that learns to squeeze data into a compact, slightly fuzzy code and rebuild it, arranged so that decoding random codes produces new, plausible examples., autoregressive image generationautoregressive image generation, n.: Making a picture one piece at a time, each piece predicted from those already made — the way chatbots write text word by word., Gumbel-softmaxGumbel-softmax, n.: A mathematical trick that lets a neural network make a choice among separate options, such as picking one word or one category, while still learning through small, smooth adjustments., image tokenizerimage tokenizer, n.: A component that chops a picture into a short sequence of 'visual words' that a language-style AI model can read or write, and turns such sequences back into pictures., Koray KavukcuogluKoray Kavukcuoglu, prop. n.: A Turkish-born computer scientist who co-authored many of DeepMind's landmark papers and in 2026 took charge of Google DeepMind as its senior vice president., neural audio codecneural audio codec, n.: A neural network that squeezes sound into a very compact stream of numbers and rebuilds it, used both to send audio cheaply and to let AI models 'write' speech and music piece by piece., Oriol VinyalsOriol Vinyals, prop. n.: A Spanish computer scientist who co-created key neural methods for translation and image captioning, led DeepMind's StarCraft-playing AlphaStar, co-led Google's Gemini models, and in 2026 co-founded the startup Discovery Loop., straight-through estimatorstraight-through estimator, n.: A shortcut that lets networks learn even when they contain steps with no useful slope, such as rounding: during learning, the computer simply pretends the step passed its input straight through., vector quantizationvector quantization, n.: Compressing data by replacing each item with the nearest entry from a small catalog of representative examples, so that only the catalog number needs to be stored or sent., VQGANVQGAN, n.: An image compressor that turns a picture into a small grid of codebook 'tiles' while keeping it sharp, used as the vocabulary of many image generators and of an early wave of AI art.