dimensionality

The number of coordinates, or independent directions, in a vector space or data representation: the length of a vectorvector, n.: An ordered list of numbers, like map coordinates. AI systems turn words, images and users into long lists of numbers so that similar things end up as nearby points., the number of featuresfeature, n.: One piece of information about each example that a model uses, like a house's size or a pixel's color, or a pattern a model has learned to detect inside itself. in a dataset, or the width of a model's hidden state (often written d_model). Embeddingembedding, n.: A list of numbers that stands for a word, picture or other item, arranged so that similar items get similar lists. It lets a computer measure similarity of meaning as distance. dimensionality trades expressiveness against memory and compute; text embeddings commonly use a few hundred to a few thousand dimensions, and Matryoshka embeddingsMatryoshka embedding, n.: An embedding built like a set of Russian nesting dolls: its first few numbers already give a rough summary, so you can cut it short to save space and still search reasonably well. can be truncated after training. High-dimensional geometry is counterintuitive: almost all the volume of a ball lies near its surface, random vectors are nearly orthogonal and distances concentrate — effects gathered under the curse of dimensionalitycurse of dimensionality, n.: The problems that pile up when each item is described by very many measurements: even huge datasets end up thinly scattered across the possibilities, and 'near' and 'far' stop meaning much.. Real data typically has a far lower intrinsic dimensionality than its ambient one, as the manifold hypothesismanifold hypothesis, n.: The idea that real-world data, such as photos of faces, fills only a thin, smoothly curved sliver of the vast space of all possible data — which is why learning from examples works at all. proposes, which motivates dimensionality reductiondimensionality reduction, n.: Squeezing data that has many measurements per item into far fewer numbers while keeping the patterns that matter, so it is easier to visualize, store or learn from..

Halving the embedding dimensionality halved index memory with little loss in recall.

Array programming

An axis of a tensortensor, n.: A block of numbers with any number of dimensions: a single number, a list, a grid, or a stack of grids. AI software stores and processes all its data this way., as in 'sum over dimension 1'. A tensor of shape (32, 512, 768) has three dimensions in this sense, while each of its 768-element vectors lives in a 768-dimensional space in the first sense.