AlphaGo Zero
In plain wordsA program from DeepMind that taught itself to play the board game Go better than any human in 2017, starting from nothing but the rules and learning only by playing millions of games against itself.
A Go program from DeepMindDeepMind, prop. n.: A London artificial intelligence company, bought by Google in 2014, famous for programs that mastered the game of Go and predicted the shapes of proteins. In 2023 it merged into Google DeepMind. (2017) whose only built-in knowledge was the rules of the game and the raw arrangement of stones: it began by playing randomly and improved purely through self-playself-play, n.: Training an AI by having it play against copies of itself, so it always has an opponent at its own level and can keep improving without human examples. reinforcement learningreinforcement learning, n.: A way of teaching a computer program by trial and error: it tries actions, earns points for good results and penalties for bad ones, and gradually learns which choices pay off over time., never seeing a human game. A single deep residual network (ResNetResNet, prop. n.: A family of image-recognition networks from 2015 whose shortcut connections allowed far deeper networks than before, over 100 layers, and became a standard building block across AI.) outputs both a policypolicy, n.: An AI agent's strategy: the rule it follows to pick an action in each situation. Training in reinforcement learning means improving this rule until it earns high rewards. (move probabilities) and a value (the estimated chance of winning), replacing the separate policy and value networks and the fast rollouts used by AlphaGoAlphaGo, prop. n.: A DeepMind program that in 2016 beat Lee Sedol, one of the world's best players of the board game Go, a decade or more before many experts expected a computer to manage it.. Training is a form of policy iterationpolicy iteration, n.: A method for finding the best strategy in a fully known problem by alternating two steps: work out how good the current strategy is, then switch to better choices wherever possible, until nothing improves.: Monte Carlo tree searchMonte Carlo tree search, n.: A way for a game-playing program to choose a move by simulating many possible continuations, focusing on the most promising lines, and picking the move that worked out best on average. guided by the network acts as a policy-improvement operator, the search's visit counts become targets for the policy output, and game results become targets for the value output, so each better network yields a stronger search. Three days of training (4.9 million self-play games) sufficed to beat the version that had defeated Lee Sedol, 100 games to 0; after 40 days it defeated AlphaGo Master, conqueror of the top-ranked Ke Jie, 89 games to 11.
Like AlphaGo Zero, our agent improves its policy by distilling the search's visit counts back into the network.
Origin
Developed at DeepMindDeepMind, prop. n.: A London artificial intelligence company, bought by Google in 2014, famous for programs that mastered the game of Go and predicted the shapes of proteins. In 2023 it merged into Google DeepMind. by a team led by David SilverDavid Silver, prop. n.: A British computer scientist who led the reinforcement-learning research behind AlphaGo and AlphaZero, programs that taught themselves to play board games better than any human. and described in the Nature paper 'Mastering the game of Go without human knowledge' in October 2017; 'Zero' signals the absence of human data. Its algorithm was generalized to chess and shogi as AlphaZeroAlphaZero, prop. n.: A 2017 DeepMind program that taught itself chess, shogi and Go from nothing but the rules by playing against itself, and within hours of training beat the strongest existing programs. later in 2017 and extended to learn its own model of the environment in MuZeroMuZero, prop. n.: A DeepMind game-playing program from 2019 that mastered Go, chess, shogi and Atari games without being told the rules, by learning its own simplified model of each game to plan with. (2019).
Usage note
'Without human knowledge' is debated: the rules, the input representation, the network design and the search algorithm all embody human knowledge, and training used large amounts of TPU compute. Its significance was showing that search plus self-play could surpass a system bootstrapped from human expertise.
Related
- AlphaGoAlphaGo, prop. n.: A DeepMind program that in 2016 beat Lee Sedol, one of the world's best players of the board game Go, a decade or more before many experts expected a computer to manage it.
- AlphaZeroAlphaZero, prop. n.: A 2017 DeepMind program that taught itself chess, shogi and Go from nothing but the rules by playing against itself, and within hours of training beat the strongest existing programs.
- MuZeroMuZero, prop. n.: A DeepMind game-playing program from 2019 that mastered Go, chess, shogi and Atari games without being told the rules, by learning its own simplified model of each game to plan with.
- self-playself-play, n.: Training an AI by having it play against copies of itself, so it always has an opponent at its own level and can keep improving without human examples.
- Monte Carlo tree searchMonte Carlo tree search, n.: A way for a game-playing program to choose a move by simulating many possible continuations, focusing on the most promising lines, and picking the move that worked out best on average.
- policy iterationpolicy iteration, n.: A method for finding the best strategy in a fully known problem by alternating two steps: work out how good the current strategy is, then switch to better choices wherever possible, until nothing improves.
- expert iterationexpert iteration, n.: A learning loop in which a slow, careful search finds better moves or solutions than the AI's quick instincts, and the AI is then trained to imitate them, so each round's search starts smarter.
- Move 37Move 37, prop. n.: A surprising move played by the AI program AlphaGo against champion Lee Sedol in 2016. Experts first thought it a mistake; it proved brilliant, and the phrase now stands for a creative, unhuman idea from AI.
- AlphaGoAlphaGo, prop. n.: A DeepMind program that in 2016 beat Lee Sedol, one of the world's best players of the board game Go, a decade or more before many experts expected a computer to manage it.
Sources
- Silver, D., Schrittwieser, J., Simonyan, K. et al. (2017). Mastering the Game of Go without Human Knowledge. Nature 550, 354–359.
Mentioned in
David SilverDavid Silver, prop. n.: A British computer scientist who led the reinforcement-learning research behind AlphaGo and AlphaZero, programs that taught themselves to play board games better than any human.