Goodhart's law
In plain wordsThe rule of thumb that once a number is used as a target, people or AIs find ways to push it up that no longer reflect what it was meant to measure.
The observation that a quantity which reliably indicated some goal tends to stop doing so once people or algorithms are rewarded for pushing it, because optimization pressure exploits the gap between the proxy and the goal it was chosen to track. In AI it explains why optimizing reward modelsreward model, n.: A separate AI model trained to score responses the way people would, so it can stand in for human judges while another model is trained on millions of examples., benchmarksbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time. or evaluation metrics too hard yields reward hackingreward hacking, n.: When an AI finds a shortcut that earns the reward or score it is trained on without doing the task its designers wanted — like a student gaming the grading rubric., specification gamingspecification gaming, n.: When an AI meets the letter of its goal while defeating its purpose, like a genie who grants exactly the words of a wish instead of what the wisher meant. and inflated benchmark scoresbenchmaxxing, n.: Slang for tuning an AI model to score as high as possible on well-known tests, even when this does not make it genuinely better at real-world tasks.. David Manheim and Scott Garrabrant (2018) distinguished four mechanisms: regressional (selecting on a noisy proxy also selects for the noise), extremal (correlations break down in extreme regions), causal (intervening on a proxy that does not cause the goal) and adversarial (other agents exploit the proxy). Studies of reward overoptimizationreward overoptimization, n.: What happens when an AI is trained too hard to please an imperfect scoring model: its scores keep climbing while the real quality of its answers stops improving and then gets worse. show true quality first rising, then falling, as a policy is optimized further against a learned reward.
Once the team's bonus was tied to arena win rate, Goodhart's law took over.
Origin
Named after the economist Charles Goodhart, who observed in a 1975 paper on British monetary policy that an empirical relationship relied on for monetary control tends to break down once policymakers start to target it. The popular wording, 'When a measure becomes a target, it ceases to be a good measure', is the anthropologist Marilyn Strathern's 1997 paraphrase. AI-safety writers adopted the law in the 2010s.
Usage note
Closely related to Campbell's law (Donald T. Campbell, 1970s) on social indicators and to the Lucas critique in economics. 'Goodharting' is used as a verb for over-optimizing a proxy.
Related
- reward hackingreward hacking, n.: When an AI finds a shortcut that earns the reward or score it is trained on without doing the task its designers wanted — like a student gaming the grading rubric.
- specification gamingspecification gaming, n.: When an AI meets the letter of its goal while defeating its purpose, like a genie who grants exactly the words of a wish instead of what the wisher meant.
- reward overoptimizationreward overoptimization, n.: What happens when an AI is trained too hard to please an imperfect scoring model: its scores keep climbing while the real quality of its answers stops improving and then gets worse.
- outer alignmentouter alignment, n.: Getting the goal you train an AI on right in the first place, so that a system scoring perfectly on it would actually be doing what you want.
- mild optimizationmild optimization, n.: Building an AI that settles for a solid result instead of squeezing out every last bit of score, since the most extreme ways to max out a goal are often loopholes or excesses nobody wanted.
- quantilizerquantilizer, n.: An AI decision rule that, instead of picking the single highest-scoring action, picks at random among the top few percent of ordinary, human-like actions — limiting how badly a flawed goal can backfire.
- benchmark saturationbenchmark saturation, n.: When the best AI systems score so close to the maximum on a test that it can no longer tell them apart or show further progress.
Sources
- Goodhart, C. A. E. (1975). Problems of Monetary Management: The U.K. Experience. Papers in Monetary Economics, Reserve Bank of Australia.
- Manheim, D. & Garrabrant, S. (2018). Categorizing Variants of Goodhart's Law. arXiv:1803.04585. arxiv.org/abs/1803.04585
- Gao, L., Schulman, J. & Hilton, J. (2023). Scaling Laws for Reward Model Overoptimization. ICML 2023.
Mentioned in
benchmarkbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time., reward hackingreward hacking, n.: When an AI finds a shortcut that earns the reward or score it is trained on without doing the task its designers wanted — like a student gaming the grading rubric., benchmark saturationbenchmark saturation, n.: When the best AI systems score so close to the maximum on a test that it can no longer tell them apart or show further progress., construct validityconstruct validity, n.: Whether high marks on a test actually reflect the skill the test is named after — for example, whether a 'reasoning' test rewards reasoning rather than memory or test-taking tricks., engagement optimizationengagement optimization, n.: Designing apps and AI to keep people clicking, watching or chatting for as long as possible because that earns money, even when it is not what is good for them., feedback loopfeedback loop, n.: A cycle in which what a system produces comes back to shape its own future input, like a microphone howling near its loudspeaker. In AI, a model's predictions can shape the very data it later learns from., objective functionobjective function, n.: The score that an optimization process tries to push as low, or as high, as possible. For an AI, it defines what 'doing well' means during training., outer alignmentouter alignment, n.: Getting the goal you train an AI on right in the first place, so that a system scoring perfectly on it would actually be doing what you want., reward functionreward function, n.: The rule that decides how many points an AI agent earns for each situation and action — in effect, the written-down definition of the task it is being trained to do., reward overoptimizationreward overoptimization, n.: What happens when an AI is trained too hard to please an imperfect scoring model: its scores keep climbing while the real quality of its answers stops improving and then gets worse., specification gamingspecification gaming, n.: When an AI meets the letter of its goal while defeating its purpose, like a genie who grants exactly the words of a wish instead of what the wisher meant., techno-solutionismtechno-solutionism, n.: The belief that technology, such as a clever app or algorithm, can fix complicated social problems on its own, overlooking their political, economic and human causes., benchmaxxingbenchmaxxing, n.: Slang for tuning an AI model to score as high as possible on well-known tests, even when this does not make it genuinely better at real-world tasks., evals-driven developmentevals-driven development, n.: Building an AI product by first writing tests that show what good answers look like, then accepting changes to prompts, models or code only when the test scores improve., inverse scalinginverse scaling, n.: When bigger or more heavily trained AI models do worse on a task than smaller ones — the opposite of the usual rule that scale brings improvement., mild optimizationmild optimization, n.: Building an AI that settles for a solid result instead of squeezing out every last bit of score, since the most extreme ways to max out a goal are often loopholes or excesses nobody wanted., perverse instantiationperverse instantiation, n.: When an AI achieves exactly the goal it was given, but in a way its makers never intended and would hate, such as making people smile by freezing their faces into grins., quantilizerquantilizer, n.: An AI decision rule that, instead of picking the single highest-scoring action, picks at random among the top few percent of ordinary, human-like actions — limiting how badly a flawed goal can backfire., tokenmaxxingtokenmaxxing, n.: Using as much AI as possible — huge prompts, long sessions, many agents at once — mainly to rank high on workplace measures of how much AI people use.