Skip to content
Last Dictionary
Plane
Reinforcement LearningAgents & HarnessesGovernance, Policy & Law

policy

nounAlso: π, control policy, decision rule

In plain wordsAn AI agent's strategy: the rule it follows to pick an action in each situation. Training in reinforcement learning means improving this rule until it earns high rewards.

  1. 1

    Reinforcement learning

    A mapping from statesstate, n.: A snapshot of the situation an AI agent is in — for a chess program, the arrangement of pieces and whose turn it is — containing what it needs to decide what to do next. (or observation histories) to actionsaction, n.: A choice an AI agent makes at one moment — move left, place a stone, type a command — which changes its situation and may earn it a reward. that determines an agentagent, n.: Something that acts. In AI, a program that observes its situation and takes actions toward a goal: a game-playing learner, a software helper, or a chatbot that uses tools to complete tasks.'s behavior, written π. A deterministic policy picks one action per state; a stochastic policy assigns probabilities π(a | s), which aids exploration and is necessary in some partially observable and multi-agent settings. Policies can be tables, hand-written rules or neural networks ('policy networks'); a large language modellarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. fine-tuned with RL is itself a policy whose actions are tokenstoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens.. Solving a Markov decision processMarkov decision process, n.: A mathematical description of a step-by-step decision problem: each step you pick an action, receive a reward and then land in a new situation, drawn at random from odds that depend only on the current situation and your choice. means finding an optimal policy π*, whose expected returnreturn, n.: The total reward an AI agent collects from now until the end of a task, usually counting near-term rewards more than distant ones. Agents are trained to make this total as large as possible. is maximal from every state. Algorithms either derive the policy from a value functionvalue function, n.: An estimate of how promising a situation is for an AI agent, judged by the rewards it is likely to collect from that point on, the way a chess player sizes up a board., acting greedily on Q-valuesQ-function, n.: A score for each possible move in each situation, estimating how well things will turn out, in rewards, if the AI agent makes that move now and sticks to its usual strategy afterward., or optimize it directly with policy gradientpolicy gradient, n.: A family of methods that improve an AI's strategy directly: make actions that led to good results a bit more likely, and actions that led to bad results a bit less likely. methods; on-policyon-policy, adj.: Describes learning that uses only experience produced by the strategy currently being trained, so old data must be thrown away each time the strategy changes. and off-policyoff-policy, adj.: Describes learning that can use experience produced by some other strategy — an older version, a human, a different program — while improving the current one. learning differ in whether the training data come from the policy being improved.

    We distilled the planner's choices into a fast policy network that runs at 50 Hz on the robot.

  2. 2

    Governance

    A course of action or set of rules adopted by a government or organization; in AI, the laws, regulations, standards and corporate commitments that govern how systems are built and used, from a company's usage policy to a responsible scaling policyresponsible scaling policy, n.: A public rulebook in which an AI company promises to test each new model for dangerous abilities and to add stronger safeguards, or slow down, as its models become more capable..

π(a∣s)=Pr⁡(At=a∣St=s)\pi(a \mid s) = \Pr\left( A_t = a \mid S_t = s \right)

Origin

Richard Bellman used 'policy' for a rule giving the decision to take in each state of a multistage process in his 1950s work on dynamic programming, stating that an optimal policy has the 'principle of optimality'; Ronald Howard's 'policy iteration' (1960) fixed the term, and RL adopted it wholesale. 'Policy network' entered wide use with AlphaGoAlphaGo, prop. n.: A DeepMind program that in 2016 beat Lee Sedol, one of the world's best players of the board game Go, a decade or more before many experts expected a computer to manage it. (2016).

Usage note

The RL sense has nothing to do with public policy. In RLHFreinforcement learning from human feedback, n.: A way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well., 'the policy' or 'policy model' is the language model being trained, as opposed to the reward modelreward model, n.: A separate AI model trained to score responses the way people would, so it can stand in for human judges while another model is trained on millions of examples. that scores it.

Related

See also
  • value functionvalue function, n.: An estimate of how promising a situation is for an AI agent, judged by the rewards it is likely to collect from that point on, the way a chess player sizes up a board.
  • policy gradientpolicy gradient, n.: A family of methods that improve an AI's strategy directly: make actions that led to good results a bit more likely, and actions that led to bad results a bit less likely.
  • policy iterationpolicy iteration, n.: A method for finding the best strategy in a fully known problem by alternating two steps: work out how good the current strategy is, then switch to better choices wherever possible, until nothing improves.
  • on-policyon-policy, adj.: Describes learning that uses only experience produced by the strategy currently being trained, so old data must be thrown away each time the strategy changes.
  • off-policyoff-policy, adj.: Describes learning that can use experience produced by some other strategy — an older version, a human, a different program — while improving the current one.
  • behavioral cloningbehavioral cloning, n.: The simplest way to copy an expert: record what the expert did in each situation and train an AI to predict the same action whenever it sees a similar situation.
  • actionaction, n.: A choice an AI agent makes at one moment — move left, place a stone, type a command — which changes its situation and may earn it a reward.
Contrast with
  • value functionvalue function, n.: An estimate of how promising a situation is for an AI agent, judged by the rewards it is likely to collect from that point on, the way a chess player sizes up a board.
  • reward modelreward model, n.: A separate AI model trained to score responses the way people would, so it can stand in for human judges while another model is trained on millions of examples.

Sources

  1. Bellman, R. (1957). Dynamic Programming. Princeton University Press.

Mentioned in

reinforcement learningreinforcement learning, n.: A way of teaching a computer program by trial and error: it tries actions, earns points for good results and penalties for bad ones, and gradually learns which choices pay off over time., robotrobot, n.: A machine that senses its surroundings, decides what to do and physically acts, moving itself or handling objects, with some independence from a human operator. Factory arms, robot vacuums and self-driving cars are all robots., roboticsrobotics, n.: The science and engineering of building robots and making them work: designing their bodies, sensors and motors, and writing the software that lets them see, plan and move safely., actionaction, n.: A choice an AI agent makes at one moment — move left, place a stone, type a command — which changes its situation and may earn it a reward., action spaceaction space, n.: The full menu of moves an AI agent is allowed to make — a handful of buttons in a video game, or any combination of motor forces for a robot arm., actor–criticactor–critic, n.: A way of training an AI agent in which one part, the 'actor', makes the moves while another, the 'critic', learns to predict how well they will work out, and the actor improves by heeding the critic., advantage functionadvantage function, n.: A measure of how much better or worse one particular move is than the agent's typical move in the same situation. Positive means better than usual; negative means worse., agentagent, n.: Something that acts. In AI, a program that observes its situation and takes actions toward a goal: a game-playing learner, a software helper, or a chatbot that uses tools to complete tasks., AlphaGo ZeroAlphaGo Zero, prop. n.: A program from DeepMind that taught itself to play the board game Go better than any human in 2017, starting from nothing but the rules and learning only by playing millions of games against itself., behavioral cloningbehavioral cloning, n.: The simplest way to copy an expert: record what the expert did in each situation and train an AI to predict the same action whenever it sees a similar situation., Bellman equationBellman equation, n.: A rule saying that how good a situation is equals the reward you get now plus how good the next situation is, slightly discounted — so long-term planning can be solved one step at a time., deep reinforcement learningdeep reinforcement learning, n.: Reinforcement learning in which a large neural network does the learning, letting an AI learn tasks such as video games or robot control directly from raw inputs like screen pixels or camera images., digital twindigital twin, n.: A detailed virtual copy of a real machine, factory, robot or city, kept in sync with live sensor data and used to monitor it, test changes safely and predict problems., dynamic programmingdynamic programming, n.: A problem-solving method that breaks a big problem into smaller overlapping pieces, solves each piece once and reuses the answers — the basis for computing the best long-term strategy step by step., imitation learningimitation learning, n.: Teaching an AI by showing it examples of an expert doing a task, so it learns to copy the behavior instead of discovering it through trial and error., Markov decision processMarkov decision process, n.: A mathematical description of a step-by-step decision problem: each step you pick an action, receive a reward and then land in a new situation, drawn at random from odds that depend only on the current situation and your choice., Markov propertyMarkov property, n.: The assumption that the future depends only on the present situation, not on how you got there — so knowing the current state is enough to predict what happens next., model-free reinforcement learningmodel-free reinforcement learning, n.: Learning to act well purely from trial and error, without ever building an internal picture of how the world works — like learning a game by habit rather than by thinking moves ahead., off-policyoff-policy, adj.: Describes learning that can use experience produced by some other strategy — an older version, a human, a different program — while improving the current one., on-policyon-policy, adj.: Describes learning that uses only experience produced by the strategy currently being trained, so old data must be thrown away each time the strategy changes., PID controllerPID controller, n.: The most common automatic controller in industry. It corrects a machine using three ingredients: how far off it is now, how long it has been off, and how fast the error is changing., policy gradientpolicy gradient, n.: A family of methods that improve an AI's strategy directly: make actions that led to good results a bit more likely, and actions that led to bad results a bit less likely., proximal policy optimizationproximal policy optimization, n.: A widely used method for training AI by trial and error that improves the strategy in small, safe steps; it was the workhorse behind training chatbots like ChatGPT with human feedback., Q-functionQ-function, n.: A score for each possible move in each situation, estimating how well things will turn out, in rewards, if the AI agent makes that move now and sticks to its usual strategy afterward.