target network
In plain wordsA slowly updated copy of an AI agent's learning network, used to compute the goals it learns toward, so those goals do not shift every time the network changes.
A copy of a value network whose parameters are held fixed, or changed only slowly, and used to compute the bootstrapped targets of temporal-differencetemporal difference learning, n.: A way of learning predictions by adjusting each guess toward the next, slightly better-informed guess instead of waiting for the final outcome — like revising a weather forecast every day as the date approaches. updates, such as r + γ max Q(s′, a′) in Q-learningQ-learning, n.: A trial-and-error method in which an agent keeps a score for every move in every situation and nudges each score toward the reward it got plus the best score available next.. Without it, the network being trained also produces its own targets, so every update moves the target as well — a feedback loop that can cause oscillation or divergence, one face of the deadly triaddeadly triad, n.: The risky combination of three common ingredients in reinforcement learning — approximation, learning from guesses, and learning from other strategies' data — which together can make learning spin out of control.. The deep Q-networkdeep Q-network, n.: A method, made famous by DeepMind in 2013–2015, in which a neural network learned to play dozens of Atari video games from screen pixels and the score alone, many at human level. copied the online weights into the target network every 10,000 updates ('hard' updates); continuous-control methods such as DDPGdeep deterministic policy gradient, n.: A method for teaching AI agents smooth, precise movements, such as controlling a robot arm, by learning a scoring network for actions and steering a decision network toward higher scores. and soft actor–criticsoft actor–critic, n.: A method for training AI agents, popular in robotics, that rewards the agent both for succeeding and for keeping its behavior varied, which helps it explore and learn reliably from reused experience. instead let the target track the online weights through an exponential moving average (Polyak averaging, with a rate such as 0.005). Double DQN also uses the target network to evaluate the action chosen by the online network, reducing overestimation. Similar slowly averaged 'target' copies appear in self-supervised methods such as BYOL.
We switched from hard target updates to Polyak averaging with τ = 0.005, and the Q-values stopped oscillating.
Origin
Introduced in DeepMind's Nature paper on the deep Q-networkdeep Q-network, n.: A method, made famous by DeepMind in 2013–2015, in which a neural network learned to play dozens of Atari video games from screen pixels and the score alone, many at human level. (Volodymyr Mnih and colleagues, 2015), which described the separate target network as a key stabilizer missing from the 2013 version; soft target updates came with DDPG (Timothy Lillicrap and colleagues, 2015).
Related
- deep Q-networkdeep Q-network, n.: A method, made famous by DeepMind in 2013–2015, in which a neural network learned to play dozens of Atari video games from screen pixels and the score alone, many at human level.
- experience replayexperience replay, n.: Storing an AI agent's past experiences in a memory bank and repeatedly learning from random samples of them, rather than only from what just happened.
- Q-learningQ-learning, n.: A trial-and-error method in which an agent keeps a score for every move in every situation and nudges each score toward the reward it got plus the best score available next.
- deadly triaddeadly triad, n.: The risky combination of three common ingredients in reinforcement learning — approximation, learning from guesses, and learning from other strategies' data — which together can make learning spin out of control.
- deep deterministic policy gradientdeep deterministic policy gradient, n.: A method for teaching AI agents smooth, precise movements, such as controlling a robot arm, by learning a scoring network for actions and steering a decision network toward higher scores.
- soft actor–criticsoft actor–critic, n.: A method for training AI agents, popular in robotics, that rewards the agent both for succeeding and for keeping its behavior varied, which helps it explore and learn reliably from reused experience.
Sources
- Mnih, V. et al. (2015). Human-level control through deep reinforcement learning. Nature 518, 529–533.
Mentioned in
deep Q-networkdeep Q-network, n.: A method, made famous by DeepMind in 2013–2015, in which a neural network learned to play dozens of Atari video games from screen pixels and the score alone, many at human level., deep reinforcement learningdeep reinforcement learning, n.: Reinforcement learning in which a large neural network does the learning, letting an AI learn tasks such as video games or robot control directly from raw inputs like screen pixels or camera images., experience replayexperience replay, n.: Storing an AI agent's past experiences in a memory bank and repeatedly learning from random samples of them, rather than only from what just happened., deadly triaddeadly triad, n.: The risky combination of three common ingredients in reinforcement learning — approximation, learning from guesses, and learning from other strategies' data — which together can make learning spin out of control., deep deterministic policy gradientdeep deterministic policy gradient, n.: A method for teaching AI agents smooth, precise movements, such as controlling a robot arm, by learning a scoring network for actions and steering a decision network toward higher scores., exponential moving average of weightsexponential moving average of weights, n.: A smoothed copy of a model kept during training, which each update nudges only slightly toward the latest weights. The calmer copy often performs better than the jittery original., soft actor–criticsoft actor–critic, n.: A method for training AI agents, popular in robotics, that rewards the agent both for succeeding and for keeping its behavior varied, which helps it explore and learn reliably from reused experience., Volodymyr MnihVolodymyr Mnih, prop. n.: A DeepMind researcher who led the 2013–2015 work in which one neural network learned to play dozens of Atari video games from screen pixels, launching deep reinforcement learning.