goal-conditioned reinforcement learning

Reinforcement learningreinforcement learning, n.: A way of teaching a computer program by trial and error: it tries actions, earns points for good results and penalties for bad ones, and gradually learns which choices pay off over time. in which the policypolicy, n.: An AI agent's strategy: the rule it follows to pick an action in each situation. Training in reinforcement learning means improving this rule until it earns high rewards. and value functionvalue function, n.: An estimate of how promising a situation is for an AI agent, judged by the rewards it is likely to collect from that point on, the way a chess player sizes up a board. take a goal as an extra input — a target position, a desired image, a language instruction — and the rewardreward, n.: The score an AI system receives after an action, telling it how good the outcome was. Reinforcement learning trains the system to collect as much reward as possible over time. measures progress toward or attainment of that goal, so that a single agent learns a whole family of tasks. Universal value function approximators generalize values across goals as well as states, letting an agent pursue goals it was never explicitly trained on. Because goal-reaching rewards are usually sparsesparse reward, n.: A situation in which an AI agent gets feedback only rarely — say, only when it finally wins — so it has almost nothing to learn from until it stumbles on success., a key technique is hindsight experience replay: a failed attempt to reach one goal is relabeled as a successful attempt to reach whatever state was actually achieved, so every trajectorytrajectory, n.: The record of what happened as an AI agent acted: the sequence of situations it was in, the actions it took and the rewards it received. becomes useful data. Goal conditioning underpins self-supervised robot learning from undirected 'play' data, hierarchical reinforcement learninghierarchical reinforcement learning, n.: Reinforcement learning organized in layers, where a high-level part chooses goals or skills — 'go to the kitchen' — and lower-level parts handle the detailed actions to achieve them. in which a manager proposes subgoals, and offline reinforcement learningoffline reinforcement learning, n.: Learning a strategy entirely from a fixed collection of recorded experience — logs, past robot runs, medical records — without trying anything new in the real world while learning. from unlabeled datasets; language-conditioned robot policies and vision-language-action modelsvision-language-action model, n.: An AI model that looks through a robot's cameras, reads an instruction such as 'put the cup in the sink', and turns both straight into the robot's movements. can be seen as goal-conditioned policies whose goals are text.

With hindsight relabeling, every failed push still taught the goal-conditioned policy how to reach somewhere.