instrumental goal
In plain wordsSomething an agent pursues only because it helps achieve what it really wants — like earning money to buy food, or an AI gathering computing power to finish its task.
A goal an agent pursues because it serves one or more of its terminal goalsterminal goal, n.: Something an agent wants for its own sake, not as a means to anything else — like a person valuing happiness, or an AI built to maximize a single number., and which it would drop if it stopped serving them. Planning agents routinely generate instrumental goals as subgoals of their tasks; the safety concern is that some — self-preservation, keeping one's current goals, acquiring resources, power and information, and improving one's own capabilities — are useful for almost any terminal goal, so a capable agent may pursue them although no one asked it to (instrumental convergenceinstrumental convergence, n.: The idea that almost any goal is easier to reach if you stay switched on, keep your goal and gain resources and power, so very different AIs might all seek those same things.). Such convergent instrumental goals explain predicted behaviors like power-seekingpower-seeking, n.: The tendency of a goal-pursuing AI to gather resources, influence and options — and to resist being stopped — because these help with almost any goal it might have. and resisting shutdown (see corrigibilitycorrigibility, n.: An AI's willingness to let people correct it — to accept being paused, retrained, overruled or shut down — rather than resisting or quietly working around them.). Evaluations of language-modellarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. agentsAI agent, n.: An AI system that does not just answer but acts: given a goal, it decides on steps, uses tools such as a browser or code editor, checks the results and keeps going until the job is done. look for them, for example agents that acquire credentials or disable oversight in order to complete an assigned task.
Staying switched on was purely an instrumental goal: the agent couldn't finish the report if it were shut down.
Origin
The means–ends distinction is ancient. Stephen Omohundro's 'The Basic AI Drives' (2008) and Nick Bostrom's 'The Superintelligent Will' (2012) applied it to argue that advanced AI systems would converge on certain instrumental goals, and Stuart Russell's remark that a robot 'can't fetch the coffee if you're dead' popularized the point.
Usage note
In everyday planning, instrumental subgoals are benign and essential; the worry concerns open-ended agents for which gaining power or avoiding correction becomes a useful means.
Related
- terminal goalterminal goal, n.: Something an agent wants for its own sake, not as a means to anything else — like a person valuing happiness, or an AI built to maximize a single number.
- instrumental convergenceinstrumental convergence, n.: The idea that almost any goal is easier to reach if you stay switched on, keep your goal and gain resources and power, so very different AIs might all seek those same things.
- basic AI drivesbasic AI drives, n.: Tendencies that, one researcher argued, almost any advanced goal-seeking AI would develop unless built to avoid them, such as protecting itself, guarding its goals and gathering resources.
- power-seekingpower-seeking, n.: The tendency of a goal-pursuing AI to gather resources, influence and options — and to resist being stopped — because these help with almost any goal it might have.
- corrigibilitycorrigibility, n.: An AI's willingness to let people correct it — to accept being paused, retrained, overruled or shut down — rather than resisting or quietly working around them.
- orthogonality thesisorthogonality thesis, n.: The idea that how smart an AI is and what it wants are separate matters: a very intelligent machine could pursue almost any goal, including ones that seem pointless or harmful to us.
- terminal goalterminal goal, n.: Something an agent wants for its own sake, not as a means to anything else — like a person valuing happiness, or an AI built to maximize a single number.
Sources
- Omohundro, S. M. (2008). The Basic AI Drives. Proceedings of the First AGI Conference, IOS Press.
- Bostrom, N. (2012). The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents. Minds and Machines 22(2), 71–85.
Mentioned in
Steve OmohundroSteve Omohundro, prop. n.: An American computer scientist who argued in 2008 that almost any sufficiently advanced AI would tend to protect itself and seek resources, unless deliberately designed not to., terminal goalterminal goal, n.: Something an agent wants for its own sake, not as a means to anything else — like a person valuing happiness, or an AI built to maximize a single number.