hierarchical reinforcement learning
In plain wordsReinforcement learning organized in layers, where a high-level part chooses goals or skills — 'go to the kitchen' — and lower-level parts handle the detailed actions to achieve them.
Reinforcement learningreinforcement learning, n.: A way of teaching a computer program by trial and error: it tries actions, earns points for good results and penalties for bad ones, and gradually learns which choices pay off over time. that decomposes behavior into several levels of temporal abstraction, so that a high-level policypolicy, n.: An AI agent's strategy: the rule it follows to pick an action in each situation. Training in reinforcement learning means improving this rule until it earns high rewards. chooses among extended skills or subgoals that lower-level policies execute over many time steps. The most influential formalism is the options framework, in which an option is a sub-policy with its own initiation set and termination condition; a Markov decision processMarkov decision process, n.: A mathematical description of a step-by-step decision problem: each step you pick an action, receive a reward and then land in a new situation, drawn at random from odds that depend only on the current situation and your choice. with options becomes a semi-MDP that standard methods can solve. Other approaches include feudal RL, in which managers set goals for workers, and MAXQ value-function decomposition. Hierarchy promises faster learning and better exploration over long horizons, reuse of skills across tasks and more interpretable behavior, but discovering useful options automatically remains difficult, and flat deep RL with large models has often matched hierarchical methods in practice. Descendants include goal-conditioned manager–worker agents, unsupervised skill discovery and large language modellarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. planners that delegate subtasks to lower-level controllers or tools.
Our hierarchical agent picks a subgoal every 25 steps and a low-level policy steers the arm toward it.
Origin
Peter Dayan and Geoffrey Hinton's 'Feudal Reinforcement Learning' (NIPS 1992) and Ronald Parr and Stuart Russell's hierarchies of abstract machines (1998) were early formulations. Richard Sutton, Doina Precup and Satinder Singh's 'Between MDPs and Semi-MDPs' (Artificial Intelligence, 1999) introduced options, and Thomas Dietterich's MAXQ (2000) decomposed value functions; Andrew Barto and Sridhar Mahadevan surveyed the field in 2003.
Related
- Markov decision processMarkov decision process, n.: A mathematical description of a step-by-step decision problem: each step you pick an action, receive a reward and then land in a new situation, drawn at random from odds that depend only on the current situation and your choice.
- policypolicy, n.: An AI agent's strategy: the rule it follows to pick an action in each situation. Training in reinforcement learning means improving this rule until it earns high rewards.
- sparse rewardsparse reward, n.: A situation in which an AI agent gets feedback only rarely — say, only when it finally wins — so it has almost nothing to learn from until it stumbles on success.
- intrinsic motivationintrinsic motivation, n.: Giving an AI agent an inner drive — such as curiosity about surprising things — so it keeps exploring and learning even when the task itself offers no reward yet.
- task decompositiontask decomposition, n.: Breaking a big, vague job into smaller, concrete steps that can each be done and checked — the way a recipe splits a meal into stages.
- plan-and-executeplan-and-execute, n.: An agent design that first writes out a complete step-by-step plan for a task, then carries out the steps one by one, revising the plan only when something goes wrong.
Sources
- Sutton, R. S., Precup, D. & Singh, S. (1999). Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence 112(1–2), 181–211.
- Dayan, P. & Hinton, G. E. (1993). Feudal Reinforcement Learning. Advances in Neural Information Processing Systems 5.
- Barto, A. G. & Mahadevan, S. (2003). Recent Advances in Hierarchical Reinforcement Learning. Discrete Event Dynamic Systems 13, 41–77.
Mentioned in
actionaction, n.: A choice an AI agent makes at one moment — move left, place a stone, type a command — which changes its situation and may earn it a reward., Richard SuttonRichard Sutton, prop. n.: A Canadian computer scientist who, with Andrew Barto, laid the groundwork for teaching machines by trial, error and reward, and whose essay 'The Bitter Lesson' argues that methods that scale with computing power always win., sparse rewardsparse reward, n.: A situation in which an AI agent gets feedback only rarely — say, only when it finally wins — so it has almost nothing to learn from until it stumbles on success., Andrew BartoAndrew Barto, prop. n.: An American computer scientist who, with his student Richard Sutton, laid the foundations of reinforcement learning, the study of how machines learn by trial and error from rewards., goal-conditioned reinforcement learninggoal-conditioned reinforcement learning, n.: Training one AI agent to reach many different goals — whichever target it is given — instead of a single fixed task, so a robot can, say, place an object anywhere asked., intrinsic motivationintrinsic motivation, n.: Giving an AI agent an inner drive — such as curiosity about surprising things — so it keeps exploring and learning even when the task itself offers no reward yet.