natural abstraction hypothesis
In plain wordsThe idea that the world naturally sorts into a limited set of useful concepts — like 'tree' or 'temperature' — that any capable mind, human or AI, would end up learning.
A conjecture by John Wentworth about why different minds might share concepts. Most details of a physical system wash out before they can affect anything far away — a rock influences distant things mainly through its mass, position and shape — so the usable information about a system is usually a compact summary; the hypothesis holds that these summaries, natural abstractions, are relatively few, that learners of very different designs, human or artificial, converge on them because prediction requires them, and that familiar human concepts are among them. If true, the hypothesis would make alignment more tractable: human concepts and values could be located inside an AI's internal representations, and interpretabilityinterpretability, n.: The effort to understand how an AI system arrives at its outputs — what information it uses and how it combines it — instead of treating it as a sealed black box. could expect recognizable featuresfeature (interpretability), n.: A concept or property that a neural network represents internally — such as 'is French', 'mentions a bridge' or 'this code has a bug' — usually corresponding to a direction in its internal activity. rather than alien ones. Supporting evidence cited includes similar features found across different networks and the 'Platonic representation hypothesis' (2024) that models' representations converge as they scale; skeptics doubt that abstractions for values, as opposed to physical objects, are natural in this sense.
If the natural abstraction hypothesis holds, 'diamond' in the model's head should mean what it means in ours.
Origin
Formulated by John Wentworth in posts on LessWrong and the AI Alignment Forum, notably 'Testing the Natural Abstraction Hypothesis: Project Intro' (2021), building on his 2020 essay 'Alignment by Default'.
Related
- universality hypothesisuniversality hypothesis, n.: The idea that different AI models trained on similar tasks tend to develop the same internal concepts and mechanisms, so what is learned about one model may carry over to others.
- linear representation hypothesislinear representation hypothesis, n.: The idea that a neural network stores many concepts as simple directions inside its internal activity, so how much of a concept is present can be read off by measuring along its direction.
- interpretabilityinterpretability, n.: The effort to understand how an AI system arrives at its outputs — what information it uses and how it combines it — instead of treating it as a sealed black box.
- ontological crisisontological crisis, n.: The puzzle an AI faces when deeper understanding leaves goals phrased in its old way of seeing things without a clear meaning — like a rule to protect 'the same person' once it learns minds can be copied.
- embedded agencyembedded agency, n.: The puzzle of how an agent should reason when it is itself part of the world it is trying to understand and change, rather than standing outside it.
- shard theoryshard theory, n.: A theory that people and AIs develop values as many small, situation-triggered habits shaped by past rewards, rather than as one fixed goal pursued single-mindedly.
Sources
- Wentworth, J. (2021). Testing the Natural Abstraction Hypothesis: Project Intro. AI Alignment Forum.
- Huh, M., Cheung, B., Wang, T. & Isola, P. (2024). The Platonic Representation Hypothesis. ICML 2024.
Mentioned in
agent foundationsagent foundations, n.: A research program that tried to work out, with mathematics, how an idealized reasoning agent should think and decide, as groundwork for building AI that stays safe., embedded agencyembedded agency, n.: The puzzle of how an agent should reason when it is itself part of the world it is trying to understand and change, rather than standing outside it., ontological crisisontological crisis, n.: The puzzle an AI faces when deeper understanding leaves goals phrased in its old way of seeing things without a clear meaning — like a rule to protect 'the same person' once it learns minds can be copied., shard theoryshard theory, n.: A theory that people and AIs develop values as many small, situation-triggered habits shaped by past rewards, rather than as one fixed goal pursued single-mindedly.