orthogonality

The relation between two vectors whose dot productdot product, n.: A score of how well two lists of numbers line up, found by multiplying matching entries and adding the products. It is large when the lists point the same way and zero when they are at right angles. is zero; in ordinary space, vectors at right angles. A set of mutually orthogonal unit vectors is orthonormal, and a square matrix with orthonormal columns is an orthogonal matrix: it rotates or reflects without stretching, preserves lengths and angles, and is perfectly conditioned. Orthogonality makes decompositions clean — the singular value decompositionsingular value decomposition, n.: A way of rewriting any grid of numbers as three simple moves in turn — turn, stretch along perpendicular axes, turn again — which reveals the directions that matter most in the data. and principal component analysisprincipal component analysis, n.: A technique that finds the few directions in which data vary the most and describes each item by its position along them, compressing many measurements into a handful of summary scores. produce orthogonal directions — and computations stable. Orthogonal weight initializationweight initialization, n.: Choosing the random starting values of a neural network's internal numbers before training. Good starting values keep signals from fading away or blowing up as they pass through many layers. helps deep and recurrent networks preserve signal, and the MuonMuon, prop. n.: A newer training method for neural networks that tidies each update so it pushes evenly in all directions within a weight table. Several groups report that it trains language models with less computing than the standard method. optimizer replaces each update matrix with an approximately orthogonal one. In high dimensions, random vectors are nearly orthogonal and exponentially many directions can be almost orthogonal, which the superpositionsuperposition, n.: The way a neural network squeezes more concepts into its internal activity than it has neurons, by letting them overlap — a main reason its insides are hard to read. hypothesis invokes to explain how networks store more featuresfeature (interpretability), n.: A concept or property that a neural network represents internally — such as 'is French', 'mentions a bridge' or 'this code has a bug' — usually corresponding to a direction in its internal activity. than they have dimensions.

We orthogonalized the steering direction against the refusal direction so that the two edits would not interfere.