MuZero
In plain wordsA DeepMind game-playing program from 2019 that mastered Go, chess, shogi and Atari games without being told the rules, by learning its own simplified model of each game to plan with.
A model-based reinforcement learningmodel-based reinforcement learning, n.: Learning in which an AI agent builds, or is given, an inner simulation of how its world responds, and uses it to plan or practice in imagination before acting. algorithm that runs Monte Carlo tree searchMonte Carlo tree search, n.: A way for a game-playing program to choose a move by simulating many possible continuations, focusing on the most promising lines, and picking the move that worked out best on average. inside a latent model it has learned for itself, rather than in a simulator supplied with the rules. Instead of predicting future observations, it learns three networks — a representation function mapping observations to a hidden state, a dynamics function predicting the next hidden state and immediate rewardreward, n.: The score an AI system receives after an action, telling it how good the outcome was. Reinforcement learning trains the system to collect as much reward as possible over time. from a hidden state and an action, and a prediction function outputting a policypolicy, n.: An AI agent's strategy: the rule it follows to pick an action in each situation. Training in reinforcement learning means improving this rule until it earns high rewards. and a valuevalue function, n.: An estimate of how promising a situation is for an AI agent, judged by the rewards it is likely to collect from that point on, the way a chess player sizes up a board. — trained end to end so that rollouts in hidden space reproduce the rewards, values and search policies actually observed. The model thus needs to capture only what matters for choosing moves and never reconstructs the board or screen. Without being told the rules, it equaled the superhuman Go, chess and shogi play of AlphaZeroAlphaZero, prop. n.: A 2017 DeepMind program that taught itself chess, shogi and Go from nothing but the rules by playing against itself, and within hours of training beat the strongest existing programs., which had been given them, and set a new aggregate record on the 57-game Atari benchmark. Descendants include Sampled MuZero for large action spaces and EfficientZero for sample efficiency; DeepMind also applied it to rate control in video compression.
Like MuZero, our planner never reconstructs pixels; it searches over learned latent states.
Origin
Introduced by Julian Schrittwieser and colleagues at DeepMind in 'Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model' (arXiv, November 2019; Nature, December 2020), as the successor to AlphaGo Zero and AlphaZero.
Related
- model-based reinforcement learningmodel-based reinforcement learning, n.: Learning in which an AI agent builds, or is given, an inner simulation of how its world responds, and uses it to plan or practice in imagination before acting.
- Monte Carlo tree searchMonte Carlo tree search, n.: A way for a game-playing program to choose a move by simulating many possible continuations, focusing on the most promising lines, and picking the move that worked out best on average.
- AlphaZeroAlphaZero, prop. n.: A 2017 DeepMind program that taught itself chess, shogi and Go from nothing but the rules by playing against itself, and within hours of training beat the strongest existing programs.
- world modelworld model, n.: An AI's internal simulation of how its world works, which lets it predict what will happen next — especially in response to its own actions — and so plan, imagine or practice without acting for real.
- self-playself-play, n.: Training an AI by having it play against copies of itself, so it always has an opponent at its own level and can keep improving without human examples.
- Arcade Learning EnvironmentArcade Learning Environment, prop. n.: A software platform that lets AI programs play classic Atari 2600 video games, used since 2012 as a standard test of how well one AI can learn many different tasks from screen images.
Sources
- Schrittwieser, J. et al. (2020). Mastering Atari, Go, chess and shogi by planning with a learned model. Nature 588, 604–609. arXiv:1911.08265. arxiv.org/abs/1911.08265
Mentioned in
world modelworld model, n.: An AI's internal simulation of how its world works, which lets it predict what will happen next — especially in response to its own actions — and so plan, imagine or practice without acting for real., AlphaGo ZeroAlphaGo Zero, prop. n.: A program from DeepMind that taught itself to play the board game Go better than any human in 2017, starting from nothing but the rules and learning only by playing millions of games against itself., AlphaZeroAlphaZero, prop. n.: A 2017 DeepMind program that taught itself chess, shogi and Go from nothing but the rules by playing against itself, and within hours of training beat the strongest existing programs., DeepMindDeepMind, prop. n.: A London artificial intelligence company, bought by Google in 2014, famous for programs that mastered the game of Go and predicted the shapes of proteins. In 2023 it merged into Google DeepMind., model-based reinforcement learningmodel-based reinforcement learning, n.: Learning in which an AI agent builds, or is given, an inner simulation of how its world responds, and uses it to plan or practice in imagination before acting., Monte Carlo tree searchMonte Carlo tree search, n.: A way for a game-playing program to choose a move by simulating many possible continuations, focusing on the most promising lines, and picking the move that worked out best on average., Arcade Learning EnvironmentArcade Learning Environment, prop. n.: A software platform that lets AI programs play classic Atari 2600 video games, used since 2012 as a standard test of how well one AI can learn many different tasks from screen images.