image tokenizer
In plain wordsA component that chops a picture into a short sequence of 'visual words' that a language-style AI model can read or write, and turns such sequences back into pictures.
A learned encoder–decoder that converts an image (or video) into a compact sequence of tokenstoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens. for a generative or multimodal model, and back again. Discrete tokenizers such as VQ-VAEVQ-VAE, n.: A network that compresses an image or sound into a short list of entries from a learned 'codebook' — like describing a picture with a fixed set of tiles — which other models can then generate. and VQGANVQGAN, n.: An image compressor that turns a picture into a small grid of codebook 'tiles' while keeping it sharp, used as the vocabulary of many image generators and of an early wave of AI art. map each cell of a downsampled grid to an index in a codebook, typically with thousands to tens of thousands of entries, so that images can be modeled by autoregressiveautoregressive image generation, n.: Making a picture one piece at a time, each piece predicted from those already made — the way chatbots write text word by word. transformers alongside text; newer quantizers (lookup-free and finite scalar quantization) allow far larger implicit vocabularies, and one-dimensional tokenizers can represent a whole image with as few as 32 tokens. Continuous tokenizers — the VAEsvariational autoencoder, n.: A neural network that learns to squeeze data into a compact, slightly fuzzy code and rebuild it, arranged so that decoding random codes produces new, plausible examples. of latent diffusionlatent diffusion model, n.: A diffusion image generator that works on a shrunken, compressed version of each picture instead of the full-size pixels, which makes it far cheaper to train and run. — output real-valued latent patches instead. Because the tokenizer bounds attainable quality and sets sequence length, designers trade compression against reconstruction fidelity and against how easily its tokens can be generated. In vision-language understanding, 'image tokens' often means continuous patch embeddingspatch embedding, n.: The first step of a vision transformer: cutting an image into small square tiles and turning each tile into a list of numbers that the model can process like a word. from a vision transformervision transformer, n.: A way of applying the transformer, the design behind chatbots, to pictures: the image is cut into small square tiles that the model reads like words in a sentence., which cannot be decoded back to pixels.
Origin
Discrete visual codes for generation date from the VQ-VAE (2017) and the discrete VAE of DALL·E (2021), with VQGAN (2020) setting the template. The term 'image tokenizer' spread with token-based generators and multimodal models such as Parti, MaskGIT and Chameleon (2022–2024).
Related
- VQ-VAEVQ-VAE, n.: A network that compresses an image or sound into a short list of entries from a learned 'codebook' — like describing a picture with a fixed set of tiles — which other models can then generate.
- VQGANVQGAN, n.: An image compressor that turns a picture into a small grid of codebook 'tiles' while keeping it sharp, used as the vocabulary of many image generators and of an early wave of AI art.
- tokenizertokenizer, n.: The component that splits text into tokens and converts them to numbers before an AI model sees it, then turns the model's output numbers back into text.
- autoregressive image generationautoregressive image generation, n.: Making a picture one piece at a time, each piece predicted from those already made — the way chatbots write text word by word.
- neural audio codecneural audio codec, n.: A neural network that squeezes sound into a very compact stream of numbers and rebuilds it, used both to send audio cheaply and to let AI models 'write' speech and music piece by piece.
- patch embeddingpatch embedding, n.: The first step of a vision transformer: cutting an image into small square tiles and turning each tile into a list of numbers that the model can process like a word.
- latent diffusion modellatent diffusion model, n.: A diffusion image generator that works on a shrunken, compressed version of each picture instead of the full-size pixels, which makes it far cheaper to train and run.
Sources
- van den Oord, A., Vinyals, O. & Kavukcuoglu, K. (2017). Neural Discrete Representation Learning. NeurIPS 2017. arXiv:1711.00937. arxiv.org/abs/1711.00937
- Esser, P., Rombach, R. & Ommer, B. (2021). Taming Transformers for High-Resolution Image Synthesis. CVPR 2021. arXiv:2012.09841. arxiv.org/abs/2012.09841
Mentioned in
tokentoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens., multimodal large language modelmultimodal large language model, n.: An AI language model that can also take in — and sometimes produce — pictures, sound or video, so you can show it a photo or speak to it, not just type., autoregressive image generationautoregressive image generation, n.: Making a picture one piece at a time, each piece predicted from those already made — the way chatbots write text word by word., neural audio codecneural audio codec, n.: A neural network that squeezes sound into a very compact stream of numbers and rebuilds it, used both to send audio cheaply and to let AI models 'write' speech and music piece by piece., patch embeddingpatch embedding, n.: The first step of a vision transformer: cutting an image into small square tiles and turning each tile into a list of numbers that the model can process like a word., vector quantizationvector quantization, n.: Compressing data by replacing each item with the nearest entry from a small catalog of representative examples, so that only the catalog number needs to be stored or sent., VQGANVQGAN, n.: An image compressor that turns a picture into a small grid of codebook 'tiles' while keeping it sharp, used as the vocabulary of many image generators and of an early wave of AI art., VQ-VAEVQ-VAE, n.: A network that compresses an image or sound into a short list of entries from a learned 'codebook' — like describing a picture with a fixed set of tiles — which other models can then generate.