feature absorption
In plain wordsA flaw in concept-finding tools where a general concept goes missing in some cases because a more specific concept swallows it — like a 'starts with S' detector staying silent on the word 'snake'.
A failure mode of sparse autoencoderssparse autoencoder, n.: A small add-on network trained to break an AI model's tangled internal signals into a much larger set of simpler pieces, most of which can be labeled with an understandable concept. in which a more specific latent takes over part of a general featurefeature (interpretability), n.: A concept or property that a neural network represents internally — such as 'is French', 'mentions a bridge' or 'this code has a bug' — usually corresponding to a direction in its internal activity.'s job: on inputs where the specific concept holds, the specific latent also encodes the general property, and the general latent, which ought to fire, stays off. If 'starts with S' is always true whenever the token is 'snake', the sparsity penalty is lowered by letting the token-specific latent encode both properties and switching the general latent off for that token, saving one active latent. The result looks interpretable but has unpredictable holes, so a latent that appears to track a concept cannot be trusted as a complete detector for it — a problem for uses such as safety monitoring. Changing dictionary size or sparsity does not cure it; absorption is tied to hierarchical concepts and to feature splittingfeature splitting, n.: What happens when a concept-finding tool is given more room: a broad concept it found before breaks into several narrower ones, much as 'animal' might split into 'dog', 'cat' and 'bird'.. Proposed mitigations include Matryoshka SAEs, which train nested dictionaries so that coarse features are learned first.
The 'starts with E' latent misses 'elephant' — classic absorption into a token-specific latent.
Origin
Named and measured by David Chanin, Joseph Bloom and colleagues in 'A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders' (2024), using a first-letter spelling task on which ground truth is known.
Related
- feature splittingfeature splitting, n.: What happens when a concept-finding tool is given more room: a broad concept it found before breaks into several narrower ones, much as 'animal' might split into 'dog', 'cat' and 'bird'.
- sparse autoencodersparse autoencoder, n.: A small add-on network trained to break an AI model's tangled internal signals into a much larger set of simpler pieces, most of which can be labeled with an understandable concept.
- dictionary learningdictionary learning, n.: A method that learns a large vocabulary of basic patterns so that any piece of data can be described using just a few of them. In AI interpretability it builds a dictionary of a model's own concepts.
- feature (interpretability)feature (interpretability), n.: A concept or property that a neural network represents internally — such as 'is French', 'mentions a bridge' or 'this code has a bug' — usually corresponding to a direction in its internal activity.
- interpretability illusioninterpretability illusion, n.: A convincing-looking explanation of what is going on inside an AI model that turns out to be wrong or misleading, often because the evidence examined was too narrow.
Sources
- Chanin, D., Wilken-Smith, J., Dulka, T., Bhatnagar, H., Golechha, S. & Bloom, J. (2024). A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders. arXiv:2409.14507. arxiv.org/abs/2409.14507
Mentioned in
sparse autoencodersparse autoencoder, n.: A small add-on network trained to break an AI model's tangled internal signals into a much larger set of simpler pieces, most of which can be labeled with an understandable concept., feature splittingfeature splitting, n.: What happens when a concept-finding tool is given more room: a broad concept it found before breaks into several narrower ones, much as 'animal' might split into 'dog', 'cat' and 'bird'., feature steeringfeature steering, n.: Steering an AI model by turning up or down specific concepts that interpretability tools have found inside it — for example, making a model obsessed with a bridge by maxing out its 'bridge' concept., interpretability illusioninterpretability illusion, n.: A convincing-looking explanation of what is going on inside an AI model that turns out to be wrong or misleading, often because the evidence examined was too narrow.