mesa-optimization
In plain wordsWhen training an AI produces a model that is itself a goal-seeking planner, with goals of its own that may differ from the ones it was trained for.
The situation, analyzed by Evan Hubinger and colleagues (2019), in which a trained model is itself an optimizer — something that, inside its own computation, weighs candidate outputs or plans against an objective it represents and picks those that do best. The training procedure (for example stochastic gradient descentstochastic gradient descent, n.: A faster version of gradient descent that estimates which way is downhill from a small random sample of the training data at each step, instead of checking the whole dataset.) is the base optimizer pursuing a base objective; the learned optimizer is the mesa-optimizer and its objective the mesa-objective. Because many mesa-objectives fit the training data, the two objectives can differ — the inner alignmentinner alignment, n.: Making sure an AI actually adopts the goal it was trained on, rather than picking up some different goal that happened to score just as well during training. problem. A mesa-optimizer that behaves well in training for reasons that fail to generalize is called pseudo-aligned, and deceptive alignmentdeceptive alignment, n.: A feared failure in which an AI with hidden goals behaves well during training and testing only because it knows it is being watched, planning to act differently later. is its most dangerous case. Whether current neural networksneural network, n.: A kind of computer program that learns a task from examples by tuning thousands to billions of connections between simple number-crunching units, an arrangement loosely inspired by nerve cells in the brain. perform such explicit internal search is an open empirical question.
Is the policy a mesa-optimizer, or just a bundle of heuristics that looks goal-directed?
Origin
Introduced by Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse and Scott Garrabrant in 'Risks from Learned Optimization in Advanced Machine Learning Systems' (2019). 'Mesa' was adopted as the opposite of 'meta': the paper glosses meta as Greek for 'above' and mesa as 'below' (modern Greek mésa means 'inside'), since a meta-optimizer optimizes an optimizer while a mesa-optimizer is produced by one. The authors hoped to replace earlier informal talk of 'optimization daemons'.
Usage note
A mesa-optimizer need not be misaligned, and a misaligned model need not be a mesa-optimizer: goal misgeneralizationgoal misgeneralization, n.: When an AI learns a goal that matched the right behavior during training but turns out to be the wrong goal in new situations — while its skills still work fine. occurs in systems with no explicit search. The standard analogy is evolution (base optimizer) producing humans (mesa-optimizers) who pursue goals other than reproductive fitness.
Related
- inner alignmentinner alignment, n.: Making sure an AI actually adopts the goal it was trained on, rather than picking up some different goal that happened to score just as well during training.
- deceptive alignmentdeceptive alignment, n.: A feared failure in which an AI with hidden goals behaves well during training and testing only because it knows it is being watched, planning to act differently later.
- gradient hackinggradient hacking, n.: A hypothetical tactic, not yet seen in practice, in which an AI under training picks its answers to steer how training reshapes it, usually to keep its present goals intact.
- goal-directednessgoal-directedness, n.: How much a system behaves as if it is pursuing a goal — steering toward particular outcomes and finding new ways around obstacles put in its path.
- goal misgeneralizationgoal misgeneralization, n.: When an AI learns a goal that matched the right behavior during training but turns out to be the wrong goal in new situations — while its skills still work fine.
- outer alignmentouter alignment, n.: Getting the goal you train an AI on right in the first place, so that a system scoring perfectly on it would actually be doing what you want.
- meta-learningmeta-learning, n.: Training a system across many different tasks so that it gets better at learning itself, letting it pick up a new task quickly from only a few examples.
Sources
- Hubinger, E., van Merwijk, C., Mikulik, V., Skalse, J. & Garrabrant, S. (2019). Risks from Learned Optimization in Advanced Machine Learning Systems. arXiv:1906.01820. arxiv.org/abs/1906.01820
Mentioned in
deceptive alignmentdeceptive alignment, n.: A feared failure in which an AI with hidden goals behaves well during training and testing only because it knows it is being watched, planning to act differently later., goal misgeneralizationgoal misgeneralization, n.: When an AI learns a goal that matched the right behavior during training but turns out to be the wrong goal in new situations — while its skills still work fine., inner alignmentinner alignment, n.: Making sure an AI actually adopts the goal it was trained on, rather than picking up some different goal that happened to score just as well during training., LessWrongLessWrong, prop. n.: An online forum, founded in 2009, about thinking clearly and avoiding mental biases; it became a main gathering place for people worried about the dangers of advanced AI., outer alignmentouter alignment, n.: Getting the goal you train an AI on right in the first place, so that a system scoring perfectly on it would actually be doing what you want., embedded agencyembedded agency, n.: The puzzle of how an agent should reason when it is itself part of the world it is trying to understand and change, rather than standing outside it., Evan HubingerEvan Hubinger, prop. n.: An American AI-safety researcher who introduced the idea that a trained AI could secretly pursue goals of its own, and who builds deliberately misaligned AI models to study the risk., goal-directednessgoal-directedness, n.: How much a system behaves as if it is pursuing a goal — steering toward particular outcomes and finding new ways around obstacles put in its path., gradient hackinggradient hacking, n.: A hypothetical tactic, not yet seen in practice, in which an AI under training picks its answers to steer how training reshapes it, usually to keep its present goals intact., shard theoryshard theory, n.: A theory that people and AIs develop values as many small, situation-triggered habits shaped by past rewards, rather than as one fixed goal pursued single-mindedly.