rank

The number of linearly independent columns of a matrixmatrix, n.: A rectangular grid of numbers arranged in rows and columns. Most of what a neural network has learned is stored in large grids like this. — always equal to the number of independent rows, and to the number of nonzero singular values in its singular value decompositionsingular value decomposition, n.: A way of rewriting any grid of numbers as three simple moves in turn — turn, stretch along perpendicular axes, turn again — which reveals the directions that matter most in the data. — and hence the dimension of the space of outputs the matrix can produce. An m × n matrix has rank at most min(m, n); it is full-rank if it reaches this bound and rank-deficient otherwise, in which case some nonzero inputs are mapped to zero and the associated linear system lacks a unique solution. A rank-r matrix can be written as a sum of r outer products, or as the product of an m × r and an r × n matrix, the basis of low-rank approximationlow-rank approximation, n.: Replacing a big grid of numbers with a slimmer one that keeps most of the same information, the way a few pencil strokes can capture a face. It saves memory and computation.. In low-rank adaptationlow-rank adaptation, n.: A cheap way to customize a big AI model: instead of changing all its billions of numbers, you train a small add-on of a few million numbers that slightly adjusts how it behaves., 'rank' is the hyperparameter r of the learned weight update; in attentionattention, n.: A way for an AI model to choose what to focus on. For each word it processes, the model scores how relevant every other word is and draws information mostly from the relevant ones., a single head's score matrix has rank at most the head dimension, a limit sometimes called the low-rank bottleneck.

We trained adapters at rank 8 and rank 64 and saw no difference on the eval.