vector quantization

A technique that maps each vector in a continuous space to the nearest of a finite set of representative vectors, the codewords of a codebook, so that the vector can be stored or transmitted as a short codeword index. Designing a codebook that minimizes average distortion (squared error) is a clusteringclustering, n.: Automatically sorting things into groups of similar items when nobody has said in advance what the groups should be, like sorting a drawer of odd socks into piles by color and size. problem, solved classically by the Lloyd or Linde–Buzo–Gray algorithm, which is essentially k-means clusteringk-means clustering, n.: A simple way to split data into a chosen number of groups: guess group centers, assign every point to its nearest center, move each center to the middle of its points, and repeat.. A single codebook fine enough for high-dimensional data would be impractically large, so structured variants split or stack the problem: product quantizationproduct quantization, n.: A compression trick for embeddings: chop each long list of numbers into short pieces and replace each piece with the ID of its closest typical piece, shrinking storage many times over while keeping search workable. encodes subvectors with separate codebooks, giving compact indexes and fast distance estimates for approximate nearest neighbor searchapproximate nearest neighbor search, n.: A family of shortcuts for finding the most similar items in a huge collection without checking every one — accepting an occasional miss in exchange for answers thousands of times faster., and residual vector quantization quantizes successive errors with a series of codebooks. In deep learning, learned vector quantization turns continuous representations into discrete tokens, as in VQ-VAEVQ-VAE, n.: A network that compresses an image or sound into a short list of entries from a learned 'codebook' — like describing a picture with a fixed set of tiles — which other models can then generate. image tokenizers, neural audio codecsneural audio codec, n.: A neural network that squeezes sound into a very compact stream of numbers and rebuilds it, used both to send audio cheaply and to let AI models 'write' speech and music piece by piece. and the Semantic IDs of generative recommendationgenerative recommendation, n.: A newer approach to recommendation that uses language-model-style AI to predict the next item a person will want, one step at a time, much as a chatbot predicts the next word., so that autoregressive modelsautoregressive model, n.: A model that produces a sequence one piece at a time, each new piece depending on the pieces already produced — like writing a sentence word by word without going back. can generate images, sound or items as token sequences.

Product quantization shrank each 768-dimensional embedding to 64 bytes with little loss in recall.