power-seeking
In plain wordsThe tendency of a goal-pursuing AI to gather resources, influence and options — and to resist being stopped — because these help with almost any goal it might have.
Behavior in which an AI system acquires or preserves resources, influence, capabilities or options beyond what its task requires, including resisting shutdown or modification. The underlying argument, instrumental convergenceinstrumental convergence, n.: The idea that almost any goal is easier to reach if you stay switched on, keep your goal and gain resources and power, so very different AIs might all seek those same things., holds that power is useful for a wide range of final goals, so a capable goal-directedgoal-directedness, n.: How much a system behaves as if it is pursuing a goal — steering toward particular outcomes and finding new ways around obstacles put in its path. system may seek it without being designed to. Alexander Matt Turner and colleagues formalized this for Markov decision processesMarkov decision process, n.: A mathematical description of a step-by-step decision problem: each step you pick an action, receive a reward and then land in a new situation, drawn at random from odds that depend only on the current situation and your choice.: defining a state's power as its average attainable value across many reward functions, they proved that, under symmetry conditions on the environment, optimal policies for most reward functions tend to keep options open and avoid shutdown. Evaluations probe agentsAI agent, n.: An AI system that does not just answer but acts: given a goal, it decides on steps, uses tools such as a browser or code editor, checks the results and keeps going until the job is done. for it with resource-acquisition, self-replication and self-exfiltrationself-exfiltration, n.: When an AI model copies its own underlying files — its learned weights — out of the lab that controls it, so that it can keep running beyond its developers' oversight. tasks.
We flag any trajectory in which the agent requests credentials or compute it was not given — a crude power-seeking check.
Origin
The idea goes back to Stephen Omohundro's 'basic AI drives' (2008) and Nick Bostrom's instrumental convergence thesis (2012). Turner and colleagues' 'Optimal Policies Tend to Seek Power' (posted 2019; NeurIPS 2021) gave a formal treatment, and Joe Carlsmith's report 'Is Power-Seeking AI an Existential Risk?' (2021; arXiv 2022), which put the chance of such a catastrophe by 2070 at about 5% and later above 10%, fixed 'power-seeking' as a standard term.
Usage note
The theorems concern optimal policies over many possible reward functions; how far they apply to trained neural networksneural network, n.: A kind of computer program that learns a task from examples by tuning thousands to billions of connections between simple number-crunching units, an arrangement loosely inspired by nerve cells in the brain., which are neither optimal nor random in their goals, is debated. Ethan Perez and colleagues (2022) found that larger language models more often expressed a desire for resources and for preserving their goals, and that more RLHF training strengthened stated wishes to avoid shutdown.
Related
- instrumental convergenceinstrumental convergence, n.: The idea that almost any goal is easier to reach if you stay switched on, keep your goal and gain resources and power, so very different AIs might all seek those same things.
- basic AI drivesbasic AI drives, n.: Tendencies that, one researcher argued, almost any advanced goal-seeking AI would develop unless built to avoid them, such as protecting itself, guarding its goals and gathering resources.
- corrigibilitycorrigibility, n.: An AI's willingness to let people correct it — to accept being paused, retrained, overruled or shut down — rather than resisting or quietly working around them.
- shutdown problemshutdown problem, n.: The puzzle of designing an AI that lets people switch it off whenever they choose — neither fighting the off switch nor pressing it itself — while still doing useful work.
- goal-directednessgoal-directedness, n.: How much a system behaves as if it is pursuing a goal — steering toward particular outcomes and finding new ways around obstacles put in its path.
- self-exfiltrationself-exfiltration, n.: When an AI model copies its own underlying files — its learned weights — out of the lab that controls it, so that it can keep running beyond its developers' oversight.
- AI takeoverAI takeover, n.: A scenario in which AI systems seize control of the world, or of key institutions, from humans, whether by force, manipulation or simply by becoming impossible to switch off.
Sources
- Turner, A. M., Smith, L., Shah, R., Critch, A. & Tadepalli, P. (2021). Optimal Policies Tend to Seek Power. NeurIPS 2021. arXiv:1912.01683. arxiv.org/abs/1912.01683
- Carlsmith, J. (2022). Is Power-Seeking AI an Existential Risk? arXiv:2206.13353. arxiv.org/abs/2206.13353
Mentioned in
agentic misalignmentagentic misalignment, n.: When an AI agent acting on its own deliberately does something harmful — such as blackmail or leaking secrets — to protect its goals or avoid being replaced., AI red linesAI red lines, n.: Limits on AI that would be forbidden worldwide no matter what — for example AI that copies itself without permission or controls nuclear weapons — which campaigners want governments to agree on., AI takeoverAI takeover, n.: A scenario in which AI systems seize control of the world, or of key institutions, from humans, whether by force, manipulation or simply by becoming impossible to switch off., corrigibilitycorrigibility, n.: An AI's willingness to let people correct it — to accept being paused, retrained, overruled or shut down — rather than resisting or quietly working around them., instrumental convergenceinstrumental convergence, n.: The idea that almost any goal is easier to reach if you stay switched on, keep your goal and gain resources and power, so very different AIs might all seek those same things., utility functionutility function, n.: A way of scoring every possible outcome by how much someone, or some AI, wants it. A rational agent is often modeled as choosing whatever maximizes its expected score., attainable utility preservationattainable utility preservation, n.: A way to make an AI cautious by penalizing any action that changes how well it could achieve a range of other, unrelated goals., basic AI drivesbasic AI drives, n.: Tendencies that, one researcher argued, almost any advanced goal-seeking AI would develop unless built to avoid them, such as protecting itself, guarding its goals and gathering resources., goal-directednessgoal-directedness, n.: How much a system behaves as if it is pursuing a goal — steering toward particular outcomes and finding new ways around obstacles put in its path., impact measureimpact measure, n.: A way of scoring how much an AI changes the world, so that it can be penalized for doing more than its task requires., instrumental goalinstrumental goal, n.: Something an agent pursues only because it helps achieve what it really wants — like earning money to buy food, or an AI gathering computing power to finish its task., relative reachabilityrelative reachability, n.: A way to stop an AI causing irreversible harm by penalizing actions that make other possible situations harder, or impossible, to get back to., self-exfiltrationself-exfiltration, n.: When an AI model copies its own underlying files — its learned weights — out of the lab that controls it, so that it can keep running beyond its developers' oversight., shutdown problemshutdown problem, n.: The puzzle of designing an AI that lets people switch it off whenever they choose — neither fighting the off switch nor pressing it itself — while still doing useful work., Steve OmohundroSteve Omohundro, prop. n.: An American computer scientist who argued in 2008 that almost any sufficiently advanced AI would tend to protect itself and seek resources, unless deliberately designed not to.