goal-directedness
In plain wordsHow much a system behaves as if it is pursuing a goal — steering toward particular outcomes and finding new ways around obstacles put in its path.
The degree to which a system's behavior is best explained and predicted as the pursuit of an objective: it steers diverse situations toward a narrow set of outcomes, adapts its means when circumstances change, and keeps doing so when obstacles are introduced. Goal-directedness matters to safety because arguments about power-seekingpower-seeking, n.: The tendency of a goal-pursuing AI to gather resources, influence and options — and to resist being stopped — because these help with almost any goal it might have. and instrumental convergenceinstrumental convergence, n.: The idea that almost any goal is easier to reach if you stay switched on, keep your goal and gain resources and power, so very different AIs might all seek those same things. apply to goal-directed systems, while a system that merely executes habits may be easier to oversee. There is no agreed definition; proposals include Daniel Dennett's intentional stance (ascribing goals when that best predicts behavior), behavioral tests of coherence and generalization, and formal measures such as maximum-entropy goal-directedness (2024). Researchers debate whether training language modelslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. as agentsAI agent, n.: An AI system that does not just answer but acts: given a goal, it decides on steps, uses tools such as a browser or code editor, checks the results and keeps going until the job is done. on long tasks makes them more goal-directed.
Before worrying about power-seeking, measure how goal-directed the policy actually is.
Origin
The idea comes from cybernetics — Arturo Rosenblueth, Norbert Wiener and Julian Bigelow's 1943 analysis of purposeful behavior — and philosophy of mind, notably Dennett's 'The Intentional Stance' (1987). In AI safety, Rohin Shah's 2018 essays questioned whether advanced AI must be goal-directed, and later work proposed quantitative measures.
Usage note
Distinct from agentagent, n.: Something that acts. In AI, a program that observes its situation and takes actions toward a goal: a game-playing learner, a software helper, or a chatbot that uses tools to complete tasks. in the software sense: an AI agentAI agent, n.: An AI system that does not just answer but acts: given a goal, it decides on steps, uses tools such as a browser or code editor, checks the results and keeps going until the job is done. may be only weakly goal-directed, and a non-agentic system can behave goal-directedly. Strongly goal-directed systems are sometimes called consequentialist or coherent.
Related
- power-seekingpower-seeking, n.: The tendency of a goal-pursuing AI to gather resources, influence and options — and to resist being stopped — because these help with almost any goal it might have.
- instrumental convergenceinstrumental convergence, n.: The idea that almost any goal is easier to reach if you stay switched on, keep your goal and gain resources and power, so very different AIs might all seek those same things.
- mesa-optimizationmesa-optimization, n.: When training an AI produces a model that is itself a goal-seeking planner, with goals of its own that may differ from the ones it was trained for.
- agentagent, n.: Something that acts. In AI, a program that observes its situation and takes actions toward a goal: a game-playing learner, a software helper, or a chatbot that uses tools to complete tasks.
- AI agentAI agent, n.: An AI system that does not just answer but acts: given a goal, it decides on steps, uses tools such as a browser or code editor, checks the results and keeps going until the job is done.
- shard theoryshard theory, n.: A theory that people and AIs develop values as many small, situation-triggered habits shaped by past rewards, rather than as one fixed goal pursued single-mindedly.
Sources
- Dennett, D. C. (1987). The Intentional Stance. MIT Press.
- MacDermott, M., Fox, J., Belardinelli, F. & Everitt, T. (2024). Measuring Goal-Directedness. arXiv:2412.04758. arxiv.org/abs/2412.04758
Mentioned in
agentic AIagentic AI, n.: AI built to pursue goals on its own — planning, taking actions and adjusting along the way — instead of only responding to one prompt at a time. 'Agentic' describes how much of this independence a system has., agentagent, n.: Something that acts. In AI, a program that observes its situation and takes actions toward a goal: a game-playing learner, a software helper, or a chatbot that uses tools to complete tasks., intentional stanceintentional stance, n.: Predicting what something will do by treating it as if it had beliefs and goals — as we do with people, and often with chess programs and chatbots — whatever is really going on inside., power-seekingpower-seeking, n.: The tendency of a goal-pursuing AI to gather resources, influence and options — and to resist being stopped — because these help with almost any goal it might have., basic AI drivesbasic AI drives, n.: Tendencies that, one researcher argued, almost any advanced goal-seeking AI would develop unless built to avoid them, such as protecting itself, guarding its goals and gathering resources., mesa-optimizationmesa-optimization, n.: When training an AI produces a model that is itself a goal-seeking planner, with goals of its own that may differ from the ones it was trained for., moral agencymoral agency, n.: The capacity to act for moral reasons and be held responsible for what one does. Researchers ask whether AI systems could ever be moral agents, or only tools whose makers and users bear responsibility., shard theoryshard theory, n.: A theory that people and AIs develop values as many small, situation-triggered habits shaped by past rewards, rather than as one fixed goal pursued single-mindedly., terminal goalterminal goal, n.: Something an agent wants for its own sake, not as a means to anything else — like a person valuing happiness, or an AI built to maximize a single number.