sparse feature circuit

A circuitcircuit, n.: A small, identifiable piece of wiring inside a neural network — a few internal parts and the connections between them — that together carry out one specific job, such as spotting curves or copying a name. whose nodes are interpretable sparse autoencodersparse autoencoder, n.: A small add-on network trained to break an AI model's tangled internal signals into a much larger set of simpler pieces, most of which can be labeled with an understandable concept. features, plus error terms, rather than neurons, attention heads or whole layers, discovered by estimating each feature's and each connection's causal effect on a task metric. Samuel Marks and colleagues computed these effects with linear approximations to activation patchingactivation patching, n.: An experiment that copies part of a model's internal state from its run on one input into its run on another, to see which internal parts actually cause a change in the answer. — attribution patchingattribution patching, n.: A fast shortcut that estimates the results of thousands of patching experiments at once, using the slope of the model's output instead of rerunning the model for every part. and an integrated-gradients variant — over SAEs trained on every layer of the model, keeping the nodes and edges whose effects exceed a threshold. Because the nodes are human-interpretable, such circuits can be inspected and edited directly. The authors introduced SHIFT (Sparse Human-Interpretable Feature Trimming): a human reviews a classifier's circuit, judges which features are irrelevant to the intended task — for example, gender-related features in a profession classifier — and ablates them, improving generalization without data that separates the two signals. Sparse feature circuits were a precursor of attribution graphsattribution graph, n.: A diagram showing which internal concepts led an AI model to a particular answer, with arrows showing how strongly each concept pushed the next., which use transcoderstranscoder, n.: A stand-in for one part of an AI model that does the same job using a large set of simple, understandable pieces, so researchers can see what that part computes. instead of SAEs to obtain cleaner connections between features.

The sparse feature circuit showed the classifier leaning on pronoun features, so we ablated them.