Skip to content
Last Dictionary
Plane
Vision, Speech & PerceptionDeep Learning

patch embedding

nounAlso: patchify, patchification, patch tokens, visual tokens, tubelet embedding

In plain wordsThe first step of a vision transformer: cutting an image into small square tiles and turning each tile into a list of numbers that the model can process like a word.

  1. The input layer of a vision transformervision transformer, n.: A way of applying the transformer, the design behind chatbots, to pictures: the image is cut into small square tiles that the model reads like words in a sentence. that converts an image into a sequence of vectors. An image of height H, width W and C channels is cut into N = HW/P² non-overlapping patches of P × P pixels; each patch is flattened and multiplied by a learned matrix E to give a D-dimensional embeddingembedding, n.: A list of numbers that stands for a word, picture or other item, arranged so that similar items get similar lists. It lets a computer measure similarity of meaning as distance., which is equivalent to a convolutionconvolution, n.: An operation that slides a small grid of weights across an image or signal, multiplying and adding at each spot, to measure how strongly a pattern appears at every location. whose kernel size and stride both equal P. Position embeddings are then added so that the arrangement of patches is not lost. Patch size sets a trade-off: smaller patches keep detail but lengthen the sequence, and attention cost grows quadratically with length (a 224 × 224 image gives 196 tokens at P = 16 but 784 at P = 8). Variants include overlapping convolutional 'stems', patch merging between stages (Swin), 3D 'tubelet' embeddings spanning several video frames, and schemes for native resolutions and aspect ratios. In vision-language modelsvision-language model, n.: An AI model that understands images and text together — it can describe a photo, answer questions about a chart or find the picture that matches a sentence., patch outputs are projected into the language model's embedding space as image tokenstoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens..

    At 448 pixels with 14-pixel patches the image costs 1,024 tokens, so we pool them before the language model.

z0=[ xclass; xp1E; xp2E; … ; xpNE ]+Epos\mathbf{z}_0 = [\,\mathbf{x}_{\text{class}};\ \mathbf{x}_p^{1}\mathbf{E};\ \mathbf{x}_p^{2}\mathbf{E};\ \dots;\ \mathbf{x}_p^{N}\mathbf{E}\,] + \mathbf{E}_{\text{pos}}

Origin

Introduced with the Vision Transformer by Alexey Dosovitskiy and colleagues at Google Brain (arXiv October 2020), whose title, 'An Image is Worth 16x16 Words', sums up the idea and whose text calls the projected patches 'patch embeddings'.

Usage note

Patch embeddings are continuous vectors; an image tokenizerimage tokenizer, n.: A component that chops a picture into a short sequence of 'visual words' that a language-style AI model can read or write, and turns such sequences back into pictures. instead maps patches to discrete codes from a learned vocabulary, typically for image generation.

Related

See also
  • vision transformervision transformer, n.: A way of applying the transformer, the design behind chatbots, to pictures: the image is cut into small square tiles that the model reads like words in a sentence.
  • embeddingembedding, n.: A list of numbers that stands for a word, picture or other item, arranged so that similar items get similar lists. It lets a computer measure similarity of meaning as distance.
  • tokentoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens.
  • positional encodingpositional encoding, n.: Extra information added to each word so a transformer knows where it sits in the text. Without it, 'dog bites man' and 'man bites dog' would look the same to the model.
  • convolutionconvolution, n.: An operation that slides a small grid of weights across an image or signal, multiplying and adding at each spot, to measure how strongly a pattern appears at every location.
  • image tokenizerimage tokenizer, n.: A component that chops a picture into a short sequence of 'visual words' that a language-style AI model can read or write, and turns such sequences back into pictures.
Contrast with
  • image tokenizerimage tokenizer, n.: A component that chops a picture into a short sequence of 'visual words' that a language-style AI model can read or write, and turns such sequences back into pictures.

Sources

  1. Dosovitskiy, A. et al. (2021). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. ICLR 2021. arxiv.org/abs/2010.11929

Mentioned in

tokentoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens., diffusion transformerdiffusion transformer, n.: A diffusion image or video generator whose core network is a transformer — the architecture behind chatbots — instead of the older image-specific design, which lets it improve more predictably as it grows., vision transformervision transformer, n.: A way of applying the transformer, the design behind chatbots, to pictures: the image is cut into small square tiles that the model reads like words in a sentence., image tokenizerimage tokenizer, n.: A component that chops a picture into a short sequence of 'visual words' that a language-style AI model can read or write, and turns such sequences back into pictures.