feature splitting
In plain wordsWhat happens when a concept-finding tool is given more room: a broad concept it found before breaks into several narrower ones, much as 'animal' might split into 'dog', 'cat' and 'bird'.
The phenomenon in which a featurefeature (interpretability), n.: A concept or property that a neural network represents internally — such as 'is French', 'mentions a bridge' or 'this code has a bug' — usually corresponding to a direction in its internal activity. learned by a small sparse autoencodersparse autoencoder, n.: A small add-on network trained to break an AI model's tangled internal signals into a much larger set of simpler pieces, most of which can be labeled with an understandable concept. dictionary is replaced, in a larger dictionary trained on the same activationsactivation, n.: The numbers a neural network computes inside itself as it processes an input — a reading of which of its artificial neurons are firing, and how strongly, at that moment., by a family of more specific features that together cover roughly the same inputs. In Anthropic's 'Towards Monosemanticity' (2023), for instance, a single base64 feature in a small dictionary split into several more specialized base64 features in larger ones, and later work found the same pattern at far larger scales. Feature splitting shows that there is no single correct number of features: the granularity of the concepts an SAE finds depends on dictionary size, and related features across sizes can be arranged into trees. It is related to feature absorptionfeature absorption, n.: A flaw in concept-finding tools where a general concept goes missing in some cases because a more specific concept swallows it — like a 'starts with S' detector staying silent on the word 'snake'., in which a general feature is left with unexplained gaps where specific features have taken over.
At 16k latents there is one math feature; at 262k it has split into algebra, geometry and LaTeX formatting.
Origin
Named and described by Trenton Bricken, Adly Templeton, Joshua Batson and colleagues in Anthropic's 'Towards Monosemanticity' (October 2023), which compared dictionaries of different sizes trained on the same one-layer transformer.
Usage note
Splitting is read by some as evidence that a model's features form a hierarchy and by others as an artifact of the sparsity objective; either way it complicates any claim to have found 'the' features of a model.
Related
- sparse autoencodersparse autoencoder, n.: A small add-on network trained to break an AI model's tangled internal signals into a much larger set of simpler pieces, most of which can be labeled with an understandable concept.
- feature absorptionfeature absorption, n.: A flaw in concept-finding tools where a general concept goes missing in some cases because a more specific concept swallows it — like a 'starts with S' detector staying silent on the word 'snake'.
- dictionary learningdictionary learning, n.: A method that learns a large vocabulary of basic patterns so that any piece of data can be described using just a few of them. In AI interpretability it builds a dictionary of a model's own concepts.
- feature (interpretability)feature (interpretability), n.: A concept or property that a neural network represents internally — such as 'is French', 'mentions a bridge' or 'this code has a bug' — usually corresponding to a direction in its internal activity.
- monosemanticitymonosemanticity, n.: The property of an internal unit of an AI model that stands for just one understandable thing, which makes it far easier to interpret than a unit that mixes several.
Sources
- Bricken, T., Templeton, A., Batson, J. et al. (2023). Towards Monosemanticity: Decomposing Language Models With Dictionary Learning. Transformer Circuits Thread.
Mentioned in
sparse autoencodersparse autoencoder, n.: A small add-on network trained to break an AI model's tangled internal signals into a much larger set of simpler pieces, most of which can be labeled with an understandable concept., enumerative safetyenumerative safety, n.: A proposed safety strategy: list every concept an AI model represents internally, then check the list for dangerous ones, such as deception, before trusting the model., feature absorptionfeature absorption, n.: A flaw in concept-finding tools where a general concept goes missing in some cases because a more specific concept swallows it — like a 'starts with S' detector staying silent on the word 'snake'., feature steeringfeature steering, n.: Steering an AI model by turning up or down specific concepts that interpretability tools have found inside it — for example, making a model obsessed with a bridge by maxing out its 'bridge' concept., parameter decompositionparameter decomposition, n.: An approach that tries to understand a neural network by splitting its weights — rather than its internal activity — into many simple pieces, each responsible for one mechanism.