decision transformer
In plain wordsA way of solving decision problems with the same kind of model that powers chatbots: it reads past situations, actions and the score it is asked to achieve, and predicts the next action.
A method that treats reinforcement learningreinforcement learning, n.: A way of teaching a computer program by trial and error: it tries actions, earns points for good results and penalties for bad ones, and gradually learns which choices pay off over time. as next-token prediction over logged experience: a transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it. that attends only to earlier positions reads trajectoriestrajectory, n.: The record of what happened as an AI agent acted: the sequence of situations it was in, the actions it took and the rewards it received. laid out as repeating triples of return-to-go (the reward still to be collected), statestate, n.: A snapshot of the situation an AI agent is in — for a chess program, the arrangement of pieces and whose turn it is — containing what it needs to decide what to do next. and actionaction, n.: A choice an AI agent makes at one moment — move left, place a stone, type a command — which changes its situation and may earn it a reward., and is trained to output each action. To act, it is given a target return, such as an expert-level score, and produces actions aimed at it, lowering the remaining target as rewards come in. Training is ordinary supervised learningsupervised learning, n.: Teaching a computer by example: you give it many inputs paired with the right answers, such as photos labeled 'cat' or 'dog', and it learns to answer correctly for new inputs., with no value functionvalue function, n.: An estimate of how promising a situation is for an AI agent, judged by the rewards it is likely to collect from that point on, the way a chess player sizes up a board., bootstrapping or policy gradientpolicy gradient, n.: A family of methods that improve an AI's strategy directly: make actions that led to good results a bit more likely, and actions that led to bad results a bit less likely., and this simplicity made it a popular baseline for offline reinforcement learningoffline reinforcement learning, n.: Learning a strategy entirely from a fixed collection of recorded experience — logs, past robot runs, medical records — without trying anything new in the real world while learning., where its 2021 results on Atari and continuous-control tasks matched or beat value-based methods. Later analyses showed that return conditioning cannot reliably stitch together the good parts of suboptimal trajectories and fails in highly stochastic environments, where a lucky high return does not indicate good actions.
Conditioning the decision transformer on a return of 90 produced noticeably better driving than conditioning on 60.
Origin
Introduced by Lili Chen, Kevin Lu and colleagues from UC Berkeley, Facebook AI Research and Google in 'Decision Transformer: Reinforcement Learning via Sequence Modeling' (arXiv, June 2021; NeurIPS 2021). Michael Janner and colleagues' Trajectory Transformer appeared the same month, and Jürgen Schmidhuber's 'upside-down reinforcement learning' (2019) had proposed reward-conditioned policies.
Related
- offline reinforcement learningoffline reinforcement learning, n.: Learning a strategy entirely from a fixed collection of recorded experience — logs, past robot runs, medical records — without trying anything new in the real world while learning.
- transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it.
- behavioral cloningbehavioral cloning, n.: The simplest way to copy an expert: record what the expert did in each situation and train an AI to predict the same action whenever it sees a similar situation.
- returnreturn, n.: The total reward an AI agent collects from now until the end of a task, usually counting near-term rewards more than distant ones. Agents are trained to make this total as large as possible.
- trajectorytrajectory, n.: The record of what happened as an AI agent acted: the sequence of situations it was in, the actions it took and the rewards it received.
- imitation learningimitation learning, n.: Teaching an AI by showing it examples of an expert doing a task, so it learns to copy the behavior instead of discovering it through trial and error.
Sources
- Chen, L. et al. (2021). Decision Transformer: Reinforcement Learning via Sequence Modeling. NeurIPS 2021. arXiv:2106.01345. arxiv.org/abs/2106.01345
- Janner, M., Li, Q. & Levine, S. (2021). Offline Reinforcement Learning as One Big Sequence Modeling Problem. NeurIPS 2021.
- Paster, K., McIlraith, S. & Ba, J. (2022). You Can't Count on Luck: Why Decision Transformers and RvS Fail in Stochastic Environments. NeurIPS 2022.
Mentioned in
offline reinforcement learningoffline reinforcement learning, n.: Learning a strategy entirely from a fixed collection of recorded experience — logs, past robot runs, medical records — without trying anything new in the real world while learning.