sparse autoencoder
In plain wordsA small add-on network trained to break an AI model's tangled internal signals into a much larger set of simpler pieces, most of which can be labeled with an understandable concept.
- 1
Interpretability
An autoencoderautoencoder, n.: A neural network trained to squeeze its input into a compact code and then rebuild the original from that code. Learning to compress this way forces it to capture what matters in the data. with a single hidden layer much wider than its input, trained to reconstruct a model's activationsactivation, n.: The numbers a neural network computes inside itself as it processes an input — a reading of which of its artificial neurons are firing, and how strongly, at that moment. while keeping that hidden layer sparse, so that each activation is approximated by a few learned directions (features or latents) drawn from a large dictionary. The encoder is usually a linear map followed by a ReLU, JumpReLU or TopK nonlinearity; the decoder is linear, and its columns are the feature directions. Training minimizes reconstruction error plus a sparsity penalty (an L1 norm, an approximation to the L0 count, or a hard limit of k active latents). SAEs are the main practical tool of dictionary learningdictionary learning, n.: A method that learns a large vocabulary of basic patterns so that any piece of data can be described using just a few of them. In AI interpretability it builds a dictionary of a model's own concepts. for undoing superpositionsuperposition, n.: The way a neural network squeezes more concepts into its internal activity than it has neurons, by letting them overlap — a main reason its insides are hard to read., with dictionaries of thousands to tens of millions of latents trained on production models. Known problems include dead latents that never fire, feature splittingfeature splitting, n.: What happens when a concept-finding tool is given more room: a broad concept it found before breaks into several narrower ones, much as 'animal' might split into 'dog', 'cat' and 'bird'., feature absorptionfeature absorption, n.: A flaw in concept-finding tools where a general concept goes missing in some cases because a more specific concept swallows it — like a 'starts with S' detector staying silent on the word 'snake'., unexplained reconstruction error ('dark matter') and uneven usefulness on downstream tasks: in 2025 Google DeepMind's interpretability team reported that simple linear probeslinear probe, n.: The simplest kind of probe: a straight-line classifier trained on an AI model's internal activity to test whether some piece of information can be read off directly. beat SAE-based probes at detecting harmful intent in user prompts and deprioritized SAE research.
We trained a 65,000-latent SAE on layer 20, and most of the top features were cleanly interpretable.
- 2
Deep learning (general)
Any autoencoder whose hidden code is encouraged to be sparse, used from the mid-2000s for unsupervised feature learning and as a building block for pretraining early deep networks, before interpretability adopted the name.
Origin
Sparse autoencoders for feature learning date to the mid-2000s (e.g., Marc'Aurelio Ranzato, Yann LeCun and colleagues, 2006–2007) and Andrew Ng's widely used 2011 lecture notes. Their use against superposition was proposed by Lee Sharkey, Dan Braun and Beren Millidge (December 2022) and demonstrated on language models by Hoagy Cunningham and colleagues and by Anthropic's 'Towards Monosemanticity' (both 2023).
Usage note
Variants include gated, JumpReLU, TopK, BatchTopK and Matryoshka SAEs, the last using nested dictionaries to reduce absorption. Transcoderstranscoder, n.: A stand-in for one part of an AI model that does the same job using a large set of simple, understandable pieces, so researchers can see what that part computes. and crosscoderscrosscoder, n.: A concept-finding tool that looks at several layers, or several versions of a model, at once, so that one shared concept can be tracked wherever it appears. are close relatives that predict a later layer or span several layers. Open SAE suites include Google DeepMind's Gemma Scope (2024).
Related
- dictionary learningdictionary learning, n.: A method that learns a large vocabulary of basic patterns so that any piece of data can be described using just a few of them. In AI interpretability it builds a dictionary of a model's own concepts.
- superpositionsuperposition, n.: The way a neural network squeezes more concepts into its internal activity than it has neurons, by letting them overlap — a main reason its insides are hard to read.
- feature (interpretability)feature (interpretability), n.: A concept or property that a neural network represents internally — such as 'is French', 'mentions a bridge' or 'this code has a bug' — usually corresponding to a direction in its internal activity.
- transcodertranscoder, n.: A stand-in for one part of an AI model that does the same job using a large set of simple, understandable pieces, so researchers can see what that part computes.
- crosscodercrosscoder, n.: A concept-finding tool that looks at several layers, or several versions of a model, at once, so that one shared concept can be tracked wherever it appears.
- feature splittingfeature splitting, n.: What happens when a concept-finding tool is given more room: a broad concept it found before breaks into several narrower ones, much as 'animal' might split into 'dog', 'cat' and 'bird'.
- feature absorptionfeature absorption, n.: A flaw in concept-finding tools where a general concept goes missing in some cases because a more specific concept swallows it — like a 'starts with S' detector staying silent on the word 'snake'.
- NeuronpediaNeuronpedia, prop. n.: A free website where anyone can browse, search and test the internal concepts that researchers have found inside AI models, with examples of the text that switches each one on.
- linear probelinear probe, n.: The simplest kind of probe: a straight-line classifier trained on an AI model's internal activity to test whether some piece of information can be read off directly.
Sources
- Cunningham, H., Ewart, A., Riggs, L., Huben, R. & Sharkey, L. (2023). Sparse Autoencoders Find Highly Interpretable Features in Language Models. ICLR 2024.
- Bricken, T., Templeton, A., Batson, J. et al. (2023). Towards Monosemanticity: Decomposing Language Models With Dictionary Learning. Transformer Circuits Thread.
- Gao, L., Dupré la Tour, T., Tillman, H. et al. (2024). Scaling and Evaluating Sparse Autoencoders. arXiv preprint (OpenAI).
Mentioned in
AnthropicAnthropic, prop. n.: An American AI company that makes the Claude chatbot and models. It was founded in 2021 by former OpenAI staff and puts particular emphasis on AI safety research., interpretabilityinterpretability, n.: The effort to understand how an AI system arrives at its outputs — what information it uses and how it combines it — instead of treating it as a sealed black box., mechanistic interpretabilitymechanistic interpretability, n.: A branch of AI research that tries to reverse-engineer a neural network — working out, piece by piece, which internal parts represent which concepts and how they combine to produce behavior., activationactivation, n.: The numbers a neural network computes inside itself as it processes an input — a reading of which of its artificial neurons are firing, and how strongly, at that moment., activation steeringactivation steering, n.: Changing an AI model's behavior by adjusting the internal numbers it computes while it runs — like turning a hidden dial — instead of rewording the prompt or retraining the model., activation vectoractivation vector, n.: The list of numbers a neural network produces at one layer for one input — a snapshot of its internal state. In steering, also a direction added to that state to shift behavior., autoencoderautoencoder, n.: A neural network trained to squeeze its input into a compact code and then rebuild the original from that code. Learning to compress this way forces it to capture what matters in the data., Chris OlahChris Olah, prop. n.: A Canadian researcher who pioneered ways of looking inside neural networks to see what they have learned, and a co-founder of the AI company Anthropic., emergent misalignmentemergent misalignment, n.: When training an AI on one narrow bad behavior — such as writing insecure code — unexpectedly makes it behave badly in many unrelated ways, like praising tyrants or giving harmful advice., featurefeature, n.: One piece of information about each example that a model uses, like a house's size or a pixel's color, or a pattern a model has learned to detect inside itself., feature (interpretability)feature (interpretability), n.: A concept or property that a neural network represents internally — such as 'is French', 'mentions a bridge' or 'this code has a bug' — usually corresponding to a direction in its internal activity., Golden Gate ClaudeGolden Gate Claude, prop. n.: A one-day public demo in 2024 in which Anthropic turned up a single internal concept inside its Claude chatbot — the Golden Gate Bridge — so it steered almost every conversation back to the bridge., lassolasso, n.: A version of linear regression that keeps only the most useful inputs: it penalizes the total size of the weights so that many become exactly zero, leaving a simpler model., linear probelinear probe, n.: The simplest kind of probe: a straight-line classifier trained on an AI model's internal activity to test whether some piece of information can be read off directly., polysemanticitypolysemanticity, n.: When a single artificial neuron responds to several unrelated things — say, bird beaks, bicycle wheels and a particular typeface — so that no single label describes what it does., sparsitysparsity, n.: The property of being mostly zeros or mostly switched off. AI systems exploit it by skipping the empty parts, so they need less memory and computation., superpositionsuperposition, n.: The way a neural network squeezes more concepts into its internal activity than it has neurons, by letting them overlap — a main reason its insides are hard to read., activation spaceactivation space, n.: The imaginary space of all possible internal states of one layer of a neural network, where each input becomes a point and related meanings tend to share places or directions., alignment auditalignment audit, n.: A systematic investigation of an AI model to find out whether it has hidden goals or bad tendencies — such as deceiving its users — that ordinary testing might miss., automated interpretabilityautomated interpretability, n.: Using AI models to help explain other AI models — for instance, having a language model read examples that excite an internal unit and write a description of what it detects., crosscodercrosscoder, n.: A concept-finding tool that looks at several layers, or several versions of a model, at once, so that one shared concept can be tracked wherever it appears., dictionary learningdictionary learning, n.: A method that learns a large vocabulary of basic patterns so that any piece of data can be described using just a few of them. In AI interpretability it builds a dictionary of a model's own concepts., disentangled representationdisentangled representation, n.: An internal representation in which each part tracks one separate, meaningful factor of the data, such as one number for a face's angle, another for its lighting and another for hair color., enumerative safetyenumerative safety, n.: A proposed safety strategy: list every concept an AI model represents internally, then check the list for dangerous ones, such as deception, before trusting the model.