imitation learning
In plain wordsTeaching an AI by showing it examples of an expert doing a task, so it learns to copy the behavior instead of discovering it through trial and error.
The family of methods that learn a policypolicy, n.: An AI agent's strategy: the rule it follows to pick an action in each situation. Training in reinforcement learning means improving this rule until it earns high rewards. from demonstrations of desired behavior — recorded trajectoriestrajectory, n.: The record of what happened as an AI agent acted: the sequence of situations it was in, the actions it took and the rewards it received. of states and actions from a human or an expert program — rather than from a rewardreward, n.: The score an AI system receives after an action, telling it how good the outcome was. Reinforcement learning trains the system to collect as much reward as possible over time. signal. The simplest approach, behavioral cloningbehavioral cloning, n.: The simplest way to copy an expert: record what the expert did in each situation and train an AI to predict the same action whenever it sees a similar situation., treats demonstrations as a supervised learningsupervised learning, n.: Teaching a computer by example: you give it many inputs paired with the right answers, such as photos labeled 'cat' or 'dog', and it learns to answer correctly for new inputs. dataset mapping observations to actions; interactive methods such as DAggerDAgger, n.: A way of teaching a robot by imitation in which an expert keeps labeling what it should have done in the situations it actually gets into, so it learns to recover from its own mistakes. query the expert on states the learner itself visits, correcting compounding errors; inverse reinforcement learninginverse reinforcement learning, n.: Working backward from behavior to goals: watching an expert act and inferring what they must be trying to achieve, so an AI can pursue the same aims. infers the reward the expert appears to optimize and then optimizes it; and adversarial imitation (GAIL) trains the policy until a discriminator cannot tell its behavior from the expert's. Imitation avoids reward design, but it is limited by the demonstrator's quality and suffers from distribution shiftdistribution shift, n.: When the data an AI system meets in real use differs from the data it was trained on — new users, places, times or habits — often making it perform worse than tests suggested. when the learner strays from demonstrated states. It is central to robot learning, where teleoperated demonstrations train diffusion policiesdiffusion policy, n.: A way of training robots in which the same kind of AI that generates images instead generates the robot's next movements, letting it copy demonstrations even when there are several good ways to do a task. and vision-language-action modelsvision-language-action model, n.: An AI model that looks through a robot's cameras, reads an instruction such as 'put the cup in the sink', and turns both straight into the robot's movements.; language-model pretraining and supervised fine-tuningsupervised fine-tuning, n.: Teaching a pretrained AI model by showing it many examples of good answers to requests, so it learns to imitate them — like learning a job by studying worked examples. are also forms of imitation.
We collected 200 teleoperated demonstrations and trained the gripper policy by imitation learning before any RL.
Origin
Learning from observed behavior was studied in 1980s robotics as 'programming by demonstration'. Dean Pomerleau's ALVINN (1989) steered a test van with a neural network, first trained on simulated road images and, from mid-1989, on a human driver's steering; Claude Sammut and colleagues' 'Learning to Fly' (1992) cloned pilots, and Pieter Abbeel and Andrew Ng's apprenticeship learning (2004) and Stéphane Ross and colleagues' DAgger (2011) gave the field theoretical foundations.
Related
- behavioral cloningbehavioral cloning, n.: The simplest way to copy an expert: record what the expert did in each situation and train an AI to predict the same action whenever it sees a similar situation.
- DAggerDAgger, n.: A way of teaching a robot by imitation in which an expert keeps labeling what it should have done in the situations it actually gets into, so it learns to recover from its own mistakes.
- inverse reinforcement learninginverse reinforcement learning, n.: Working backward from behavior to goals: watching an expert act and inferring what they must be trying to achieve, so an AI can pursue the same aims.
- offline reinforcement learningoffline reinforcement learning, n.: Learning a strategy entirely from a fixed collection of recorded experience — logs, past robot runs, medical records — without trying anything new in the real world while learning.
- teleoperationteleoperation, n.: Controlling a robot or machine from a distance, with a person steering it in real time using a joystick, a matching 'puppet' arm, a VR headset or a screen.
- diffusion policydiffusion policy, n.: A way of training robots in which the same kind of AI that generates images instead generates the robot's next movements, letting it copy demonstrations even when there are several good ways to do a task.
- supervised fine-tuningsupervised fine-tuning, n.: Teaching a pretrained AI model by showing it many examples of good answers to requests, so it learns to imitate them — like learning a job by studying worked examples.
- reinforcement learningreinforcement learning, n.: A way of teaching a computer program by trial and error: it tries actions, earns points for good results and penalties for bad ones, and gradually learns which choices pay off over time.
Sources
- Pomerleau, D. A. (1989). ALVINN: An Autonomous Land Vehicle in a Neural Network. Advances in Neural Information Processing Systems 1.
- Osa, T. et al. (2018). An Algorithmic Perspective on Imitation Learning. Foundations and Trends in Robotics 7(1–2), 1–179.
Mentioned in
humanoid robothumanoid robot, n.: A robot built roughly in human shape, with a head, torso, two arms and usually two legs, so that it can work in spaces and with tools designed for people., reinforcement learningreinforcement learning, n.: A way of teaching a computer program by trial and error: it tries actions, earns points for good results and penalties for bad ones, and gradually learns which choices pay off over time., behavioral cloningbehavioral cloning, n.: The simplest way to copy an expert: record what the expert did in each situation and train an AI to predict the same action whenever it sees a similar situation., manipulationmanipulation, n.: A robot's use of arms, hands or grippers to change the world around it: picking things up, moving, pushing, opening, assembling or using tools., robot learningrobot learning, n.: Teaching robots skills from data and experience, such as watching demonstrations, practicing by trial and error or training in simulation, instead of programming every movement by hand., sparse rewardsparse reward, n.: A situation in which an AI agent gets feedback only rarely — say, only when it finally wins — so it has almost nothing to learn from until it stumbles on success., supervised fine-tuningsupervised fine-tuning, n.: Teaching a pretrained AI model by showing it many examples of good answers to requests, so it learns to imitate them — like learning a job by studying worked examples., teleoperationteleoperation, n.: Controlling a robot or machine from a distance, with a person steering it in real time using a joystick, a matching 'puppet' arm, a VR headset or a screen., trajectorytrajectory, n.: The record of what happened as an AI agent acted: the sequence of situations it was in, the actions it took and the rewards it received., action chunkingaction chunking, n.: Having a robot's AI plan a short burst of upcoming movements, say the next second, all at once instead of one tiny motion at a time, which makes motion smoother and mistakes rarer., AlphaStarAlphaStar, prop. n.: A DeepMind AI that learned to play the real-time strategy game StarCraft II well enough to reach its highest public rank, Grandmaster, in 2019., CoRLCoRL, prop. n.: A yearly academic conference, started in 2017, for research where robotics meets machine learning., DAggerDAgger, n.: A way of teaching a robot by imitation in which an expert keeps labeling what it should have done in the situations it actually gets into, so it learns to recover from its own mistakes., decision transformerdecision transformer, n.: A way of solving decision problems with the same kind of model that powers chatbots: it reads past situations, actions and the score it is asked to achieve, and predicts the next action., diffusion policydiffusion policy, n.: A way of training robots in which the same kind of AI that generates images instead generates the robot's next movements, letting it copy demonstrations even when there are several good ways to do a task., expert iterationexpert iteration, n.: A learning loop in which a slow, careful search finds better moves or solutions than the AI's quick instincts, and the AI is then trained to imitate them, so each round's search starts smarter., exposure biasexposure bias, n.: A weakness of models trained only on perfect examples: when they write on their own, one early mistake puts them in unfamiliar territory they never practiced, and errors can snowball., inverse reinforcement learninginverse reinforcement learning, n.: Working backward from behavior to goals: watching an expert act and inferring what they must be trying to achieve, so an AI can pursue the same aims., offline reinforcement learningoffline reinforcement learning, n.: Learning a strategy entirely from a fixed collection of recorded experience — logs, past robot runs, medical records — without trying anything new in the real world while learning., on-policy distillationon-policy distillation, n.: A way of teaching a small AI from a bigger one where the small AI writes its own answers and the bigger one grades every word, so the learner is corrected on its own mistakes., Oriol VinyalsOriol Vinyals, prop. n.: A Spanish computer scientist who co-created key neural methods for translation and image captioning, led DeepMind's StarCraft-playing AlphaStar, co-led Google's Gemini models, and in 2026 co-founded the startup Discovery Loop., Pieter AbbeelPieter Abbeel, prop. n.: A Belgian-American roboticist at Berkeley whose research taught robots to learn from demonstrations and from trial and error, and who co-founded the robotics company Covariant., quantilizerquantilizer, n.: An AI decision rule that, instead of picking the single highest-scoring action, picks at random among the top few percent of ordinary, human-like actions — limiting how badly a flawed goal can backfire.