covariance

The expected valueexpected value, n.: The long-run average you would get if a chancy situation could be repeated many times — 3.5 for a fair die, even though no single roll shows 3.5. of the product of two random variablesrandom variable, n.: A quantity whose value depends on chance, such as the number rolled on a die or the next word a chatbot will write.' deviations from their means: positive when they tend to be above their means together, negative when one tends to be high while the other is low, and zero for independent variables (though zero covariance does not imply independence). Dividing by both standard deviations gives Pearson correlationcorrelation, n.: A measure of how closely two quantities move together, from −1 (one rises as the other falls) through 0 (no straight-line link) to +1 (they rise together). It does not show that one causes the other.. For a random vector, the covariance matrix collects all pairwise covariances, with variancesvariance, n.: A number that grows the more widely something's possible values scatter away from its average, and is zero if it never varies. Its square root, the standard deviation, uses the same units as the values themselves. on its diagonal; it is symmetric and positive semidefinite, sets the shape of a multivariate normal distributionnormal distribution, n.: The familiar bell-shaped curve: most values cluster near the average, and values become rapidly rarer the farther they lie from it. Measurement errors and adult heights roughly follow it., and its eigenvectorseigenvector, n.: A special direction for a transformation: anything pointing that way stays on its line, only longer, shorter or reversed; the amount of stretch is the eigenvalue. Together they reveal the main axes along which data or a system varies. are the axes found by principal component analysisprincipal component analysis, n.: A technique that finds the few directions in which data vary the most and describes each item by its position along them, compressing many measurements into a handful of summary scores.. Covariance matrices appear as kernels in Gaussian processesGaussian process, n.: A way of predicting an unknown curve or surface that gives not just a best guess at each point but an honest error bar, narrow near measured points and wide far from them., in Kalman filtersKalman filter, n.: A method that keeps a running best guess of something you cannot measure exactly, such as a vehicle's position, by blending predictions from a motion model with noisy sensor readings., in the Fréchet inception distanceFréchet inception distance, n.: A score for image generators that compares the overall statistics of their pictures with those of real photos, as seen by an image-recognition network; lower is better., in second-order optimizers that approximate gradient covariance, and in whitening of data and activationsactivation, n.: The numbers a neural network computes inside itself as it processes an input — a reading of which of its artificial neurons are firing, and how strongly, at that moment..

Whitening the embeddings with their covariance matrix spread out the cosine similarities considerably.