polysemanticity
In plain wordsWhen a single artificial neuron responds to several unrelated things — say, bird beaks, bicycle wheels and a particular typeface — so that no single label describes what it does.
The property of a network unit, usually a neuron, that activates for several unrelated featuresfeature (interpretability), n.: A concept or property that a neural network represents internally — such as 'is French', 'mentions a bridge' or 'this code has a bug' — usually corresponding to a direction in its internal activity.: Anthropic's 2023 monosemanticity study, for example, described one neuron in a small language model responding to academic citations, English dialogue, HTTP requests and Korean text. Polysemanticity is the main obstacle to interpreting networks neuron by neuron. The leading explanation is superpositionsuperposition, n.: The way a neural network squeezes more concepts into its internal activity than it has neurons, by letting them overlap — a main reason its insides are hard to read.: when a model must represent more features than it has neurons, features are spread across overlapping directions, so each neuron takes part in many. Polysemanticity can also arise incidentally from training dynamics even when capacity would suffice. Remedies either change the model — architectures or penalties that encourage monosemanticitymonosemanticity, n.: The property of an internal unit of an AI model that stands for just one understandable thing, which makes it far easier to interpret than a unit that mixes several., such as the softmax linear unit — or decompose its activations after training with dictionary learningdictionary learning, n.: A method that learns a large vocabulary of basic patterns so that any piece of data can be described using just a few of them. In AI interpretability it builds a dictionary of a model's own concepts., whose features are usually far more interpretable than neurons.
That neuron is hopelessly polysemantic; look at the SAE features instead.
Origin
Borrowed from linguistics, where polysemy means one word having several meanings. 'Polysemantic neurons' were described in Chris Olah and colleagues' 'Zoom In' (Distill, March 2020), which opened the Circuits work on image models, and became a central concern after Anthropic's studies of superposition (2022) and monosemanticity (2023).
Usage note
The closest neuroscience analogue is mixed selectivity: neurons tuned to combinations of task variables. Polysemanticity is a property of units, not of the features themselves.
Related
- superpositionsuperposition, n.: The way a neural network squeezes more concepts into its internal activity than it has neurons, by letting them overlap — a main reason its insides are hard to read.
- feature (interpretability)feature (interpretability), n.: A concept or property that a neural network represents internally — such as 'is French', 'mentions a bridge' or 'this code has a bug' — usually corresponding to a direction in its internal activity.
- sparse autoencodersparse autoencoder, n.: A small add-on network trained to break an AI model's tangled internal signals into a much larger set of simpler pieces, most of which can be labeled with an understandable concept.
- dictionary learningdictionary learning, n.: A method that learns a large vocabulary of basic patterns so that any piece of data can be described using just a few of them. In AI interpretability it builds a dictionary of a model's own concepts.
- privileged basisprivileged basis, n.: A property of a layer in a neural network where the individual numbered slots, not just blends of them, carry special meaning, because something in the design treats each slot separately.
- artificial neuronartificial neuron, n.: The basic building block of a neural network: a tiny calculator that multiplies each incoming number by a weight, adds them up, and passes the total through a simple function to decide its output.
- monosemanticitymonosemanticity, n.: The property of an internal unit of an AI model that stands for just one understandable thing, which makes it far easier to interpret than a unit that mixes several.
Sources
- Olah, C., Cammarata, N., Schubert, L., Goh, G., Petrov, M. & Carter, S. (2020). Zoom In: An Introduction to Circuits. Distill. distill.pub/2020/circuits/zoom-in
- Lecomte, V., Thaman, K., Schaeffer, R., Bashkansky, N., Chow, T. & Koyejo, S. (2024). What Causes Polysemanticity? An Alternative Origin Story of Mixed Selectivity from Incidental Causes. arXiv preprint.
Mentioned in
artificial neuronartificial neuron, n.: The basic building block of a neural network: a tiny calculator that multiplies each incoming number by a weight, adds them up, and passes the total through a simple function to decide its output., feature (interpretability)feature (interpretability), n.: A concept or property that a neural network represents internally — such as 'is French', 'mentions a bridge' or 'this code has a bug' — usually corresponding to a direction in its internal activity., superpositionsuperposition, n.: The way a neural network squeezes more concepts into its internal activity than it has neurons, by letting them overlap — a main reason its insides are hard to read., feature visualizationfeature visualization, n.: Making pictures that show what a part of an image-recognition network is looking for, by repeatedly adjusting an image until that part responds as strongly as possible., knowledge neuronknowledge neuron, n.: An internal unit of a language model claimed to be responsible for storing a particular fact, such that turning it up or down makes the model more or less likely to state that fact., monosemanticitymonosemanticity, n.: The property of an internal unit of an AI model that stands for just one understandable thing, which makes it far easier to interpret than a unit that mixes several., multimodal neuronmultimodal neuron, n.: A unit inside an AI vision model that responds to one idea however it appears — a photo, a drawing or the written name — much like the 'Halle Berry neuron' found in a human brain., network dissectionnetwork dissection, n.: A method for finding out what individual units inside an image-recognition network respond to, such as dogs, windows, grass or stripes, by comparing where each unit lights up with pictures labeled by people., privileged basisprivileged basis, n.: A property of a layer in a neural network where the individual numbered slots, not just blends of them, carry special meaning, because something in the design treats each slot separately.