trajectory
In plain wordsThe record of what happened as an AI agent acted: the sequence of situations it was in, the actions it took and the rewards it received.
- 1
Reinforcement learning
A sequence of statesstate, n.: A snapshot of the situation an AI agent is in — for a chess program, the arrangement of pieces and whose turn it is — containing what it needs to decide what to do next., actionsaction, n.: A choice an AI agent makes at one moment — move left, place a stone, type a command — which changes its situation and may earn it a reward. and rewardsreward, n.: The score an AI system receives after an action, telling it how good the outcome was. Reinforcement learning trains the system to collect as much reward as possible over time., τ = (s₀, a₀, r₁, s₁, a₁, r₂, …), generated by running a policypolicy, n.: An AI agent's strategy: the rule it follows to pick an action in each situation. Training in reinforcement learning means improving this rule until it earns high rewards. in an environmentenvironment, n.: Everything an AI agent interacts with but does not control — a game, a simulator, a website, the physical world. It answers each of the agent's actions with a new situation and, in reinforcement learning, a score.: either a complete episodeepisode, n.: One complete run of a task from start to finish — a single game of chess, one attempt at a maze — after which everything resets and the agent tries again. or a segment of one. Trajectories are the raw data of reinforcement learning: policy gradientpolicy gradient, n.: A family of methods that improve an AI's strategy directly: make actions that led to good results a bit more likely, and actions that led to bad results a bit less likely. methods weight the log-probabilities of each trajectory's actions by its returnreturn, n.: The total reward an AI agent collects from now until the end of a task, usually counting near-term rewards more than distant ones. Agents are trained to make this total as large as possible., off-policy methods store its transitions in a replay bufferexperience replay, n.: Storing an AI agent's past experiences in a memory bank and repeatedly learning from random samples of them, rather than only from what just happened., and imitation learningimitation learning, n.: Teaching an AI by showing it examples of an expert doing a task, so it learns to copy the behavior instead of discovering it through trial and error. trains on expert trajectories. In language-model training a trajectory comprises the prompt, the generated reasoning and answer, any tool callstool call, n.: A single request from an AI model to use one of its tools — naming the tool and filling in the details — which the surrounding software then carries out and reports back on. and their results, and the final reward; agent trajectories are also read by engineers to diagnose failures.
We sampled 64 trajectories per prompt and kept only those that ended with a passing test.
- 2
Robotics and control
A path through state space as a function of time — the positions, velocities and often forces a robot or vehicle should follow — computed by a planner or by trajectory optimizationtrajectory optimization, n.: Calculating the best way for a robot to move over time, choosing exact positions, speeds and forces moment by moment so the motion is smooth, fast or energy-efficient and avoids obstacles. and tracked by a feedback controller.
Origin
From the Latin trajectus, 'thrown across'; long used in mechanics for the path of a projectile and in control theory for a system's path through state space. RL adopted the word for sampled interaction sequences, and the symbol τ became common in policy-gradient papers of the 2010s.
Related
- episodeepisode, n.: One complete run of a task from start to finish — a single game of chess, one attempt at a maze — after which everything resets and the agent tries again.
- rolloutrollout, n.: Letting an AI play out a task with its current strategy, from some starting point to the end, to see how things turn out and gather data for learning or planning.
- returnreturn, n.: The total reward an AI agent collects from now until the end of a task, usually counting near-term rewards more than distant ones. Agents are trained to make this total as large as possible.
- experience replayexperience replay, n.: Storing an AI agent's past experiences in a memory bank and repeatedly learning from random samples of them, rather than only from what just happened.
- policy gradientpolicy gradient, n.: A family of methods that improve an AI's strategy directly: make actions that led to good results a bit more likely, and actions that led to bad results a bit less likely.
- trajectory optimizationtrajectory optimization, n.: Calculating the best way for a robot to move over time, choosing exact positions, speeds and forces moment by moment so the motion is smooth, fast or energy-efficient and avoids obstacles.
Mentioned in
episodeepisode, n.: One complete run of a task from start to finish — a single game of chess, one attempt at a maze — after which everything resets and the agent tries again., imitation learningimitation learning, n.: Teaching an AI by showing it examples of an expert doing a task, so it learns to copy the behavior instead of discovering it through trial and error., model-based reinforcement learningmodel-based reinforcement learning, n.: Learning in which an AI agent builds, or is given, an inner simulation of how its world responds, and uses it to plan or practice in imagination before acting., policy gradientpolicy gradient, n.: A family of methods that improve an AI's strategy directly: make actions that led to good results a bit more likely, and actions that led to bad results a bit less likely., returnreturn, n.: The total reward an AI agent collects from now until the end of a task, usually counting near-term rewards more than distant ones. Agents are trained to make this total as large as possible., rolloutrollout, n.: Letting an AI play out a task with its current strategy, from some starting point to the end, to see how things turn out and gather data for learning or planning., decision transformerdecision transformer, n.: A way of solving decision problems with the same kind of model that powers chatbots: it reads past situations, actions and the score it is asked to achieve, and predicts the next action., episodic memoryepisodic memory, n.: Memory of particular past events — what happened, when and in what order. AI agents keep records of earlier conversations or attempts so they can recall them and learn from them., generalized advantage estimationgeneralized advantage estimation, n.: A recipe for judging how much better each action was than expected by blending short-term and long-term evidence, trading a little accuracy for much steadier learning., goal-conditioned reinforcement learninggoal-conditioned reinforcement learning, n.: Training one AI agent to reach many different goals — whichever target it is given — instead of a single fixed task, so a robot can, say, place an object anywhere asked.