inverse reinforcement learning
In plain wordsWorking backward from behavior to goals: watching an expert act and inferring what they must be trying to achieve, so an AI can pursue the same aims.
The problem of recovering a reward functionreward function, n.: The rule that decides how many points an AI agent earns for each situation and action — in effect, the written-down definition of the task it is being trained to do. that explains observed behavior, assumed to be approximately optimal, in a known environmentenvironment, n.: Everything an AI agent interacts with but does not control — a game, a simulator, a website, the physical world. It answers each of the agent's actions with a new situation and, in reinforcement learning, a score. — the reverse of reinforcement learningreinforcement learning, n.: A way of teaching a computer program by trial and error: it tries actions, earns points for good results and penalties for bad ones, and gradually learns which choices pay off over time., which derives behavior from a given reward. The problem is ill-posed, since many rewards (even a constant one) make the same behavior optimal, so methods add assumptions: matching the expected features of expert trajectories (apprenticeship learning), choosing the maximum-entropy distribution over trajectories consistent with those features, placing Bayesian priors over rewards, or training adversarially so that a discriminator's output serves as a learned reward (GAIL, AIRL). Recovered rewards can transfer to new situations better than copied actions, and they make inferred goals inspectable. IRL is used to model drivers, pedestrians and robot users, and is a conceptual foundation of value learningvalue learning, n.: Teaching an AI what people want by having it learn from their behavior, choices and feedback, rather than trying to write every goal down in advance. in AI safety — learning what people want from what they do — elaborated in cooperative inverse reinforcement learningcooperative inverse reinforcement learning, n.: A model of a human and an AI working as a team, where only the human knows the goal and the AI must work it out while helping.; learning a reward modelreward model, n.: A separate AI model trained to score responses the way people would, so it can stand in for human judges while another model is trained on millions of examples. from human preferences is a close relative.
From hours of taxi trajectories, inverse reinforcement learning recovered how drivers trade travel time against tolls.
Origin
Rudolf Kálmán's 1964 question of when a linear control system is optimal founded inverse optimal control. Stuart Russell posed inverse reinforcement learning for AI in a 1998 COLT paper, and Andrew Ng and Russell's 'Algorithms for Inverse Reinforcement Learning' (ICML 2000) gave the first methods; Pieter Abbeel and Ng (2004) and Brian Ziebart and colleagues (2008) followed.
Related
- imitation learningimitation learning, n.: Teaching an AI by showing it examples of an expert doing a task, so it learns to copy the behavior instead of discovering it through trial and error.
- reward functionreward function, n.: The rule that decides how many points an AI agent earns for each situation and action — in effect, the written-down definition of the task it is being trained to do.
- value learningvalue learning, n.: Teaching an AI what people want by having it learn from their behavior, choices and feedback, rather than trying to write every goal down in advance.
- cooperative inverse reinforcement learningcooperative inverse reinforcement learning, n.: A model of a human and an AI working as a team, where only the human knows the goal and the AI must work it out while helping.
- reward modelreward model, n.: A separate AI model trained to score responses the way people would, so it can stand in for human judges while another model is trained on millions of examples.
- behavioral cloningbehavioral cloning, n.: The simplest way to copy an expert: record what the expert did in each situation and train an AI to predict the same action whenever it sees a similar situation.
- reinforcement learningreinforcement learning, n.: A way of teaching a computer program by trial and error: it tries actions, earns points for good results and penalties for bad ones, and gradually learns which choices pay off over time.
Sources
- Ng, A. Y. & Russell, S. (2000). Algorithms for Inverse Reinforcement Learning. Proceedings of ICML 2000.
- Russell, S. (1998). Learning agents for uncertain environments (extended abstract). Proceedings of COLT 1998.
- Ziebart, B. D., Maas, A., Bagnell, J. A. & Dey, A. K. (2008). Maximum Entropy Inverse Reinforcement Learning. AAAI 2008.
Mentioned in
behavioral cloningbehavioral cloning, n.: The simplest way to copy an expert: record what the expert did in each situation and train an AI to predict the same action whenever it sees a similar situation., imitation learningimitation learning, n.: Teaching an AI by showing it examples of an expert doing a task, so it learns to copy the behavior instead of discovering it through trial and error., reward functionreward function, n.: The rule that decides how many points an AI agent earns for each situation and action — in effect, the written-down definition of the task it is being trained to do., Stuart RussellStuart Russell, prop. n.: A British computer scientist who co-wrote the standard AI textbook and argues that AI must be redesigned to stay uncertain about what humans want, so that it remains under human control., value learningvalue learning, n.: Teaching an AI what people want by having it learn from their behavior, choices and feedback, rather than trying to write every goal down in advance., Center for Human-Compatible AICenter for Human-Compatible AI, prop. n.: Stuart Russell's group at the University of California, Berkeley, which argues that AI should be built to be unsure of what people want and to learn it by watching them., cooperative inverse reinforcement learningcooperative inverse reinforcement learning, n.: A model of a human and an AI working as a team, where only the human knows the goal and the AI must work it out while helping., inverse reward designinverse reward design, n.: A method in which an AI treats the reward its designer wrote as a clue to what the designer wanted, rather than as a perfect statement of the goal., Pieter AbbeelPieter Abbeel, prop. n.: A Belgian-American roboticist at Berkeley whose research taught robots to learn from demonstrations and from trial and error, and who co-founded the robotics company Covariant.