multi-agent reinforcement learning
In plain wordsReinforcement learning with several AI agents learning at once in a shared world, where each must adapt to the others — as teammates, rivals or both.
Reinforcement learningreinforcement learning, n.: A way of teaching a computer program by trial and error: it tries actions, earns points for good results and penalties for bad ones, and gradually learns which choices pay off over time. in which several agentsagent, n.: Something that acts. In AI, a program that observes its situation and takes actions toward a goal: a game-playing learner, a software helper, or a chatbot that uses tools to complete tasks. act and learn in a shared environmentenvironment, n.: Everything an AI agent interacts with but does not control — a game, a simulator, a website, the physical world. It answers each of the agent's actions with a new situation and, in reinforcement learning, a score., usually modeled as a stochastic (Markov) game: each agent has its own observations, actions and rewards, and state transitions depend on all of their actions. Settings range from fully cooperative (a shared reward, as in robot teams) through mixed general-sum interactions such as negotiation and traffic to fully competitive zero-sum games. Other learners make each agent's environment non-stationary, break single-agent convergence guarantees and raise problems of equilibrium selection, coordination, communication and credit assignment among teammates. Common approaches include independent learners, centralized training with decentralized execution (centralized critics, value decomposition), self-playself-play, n.: Training an AI by having it play against copies of itself, so it always has an opponent at its own level and can keep improving without human examples. and population-based training, and game-theoretic methods such as counterfactual regret minimizationcounterfactual regret minimization, n.: A method for working out near-unbeatable strategies in games with hidden information, like poker, by repeatedly playing against itself and shifting each decision toward the moves it most regrets not making.. Landmarks include superhuman poker programs, AlphaStar in StarCraft II, OpenAI Five in Dota 2 and Meta's Cicero in the negotiation game Diplomacy.
Each drone learns its own policy, but a centralized critic sees the whole swarm during training.
Origin
Lloyd Shapley's stochastic games (1953) supplied the formal model. Ming Tan's 'Multi-Agent Reinforcement Learning: Independent vs. Cooperative Agents' (ICML 1993) and Michael Littman's 'Markov Games as a Framework for Multi-Agent Reinforcement Learning' (ICML 1994) founded the field; deep MARL grew from the mid-2010s with algorithms such as MADDPG (2017) and QMIX (2018).
Usage note
Distinct from a multi-agent systemmulti-agent system, n.: A setup in which several AI agents, each with its own role or point of view, work together — or compete — on a problem, passing messages and results among themselves. of language-model agents coordinated through prompts and tools, which usually involves no joint learning, although the two fields increasingly overlap.
Related
- self-playself-play, n.: Training an AI by having it play against copies of itself, so it always has an opponent at its own level and can keep improving without human examples.
- counterfactual regret minimizationcounterfactual regret minimization, n.: A method for working out near-unbeatable strategies in games with hidden information, like poker, by repeatedly playing against itself and shifting each decision toward the moves it most regrets not making.
- game treegame tree, n.: A branching diagram of every way a game could unfold: each point is a position, each branch a possible move, and the tips of the branches are wins, losses or draws.
- Markov decision processMarkov decision process, n.: A mathematical description of a step-by-step decision problem: each step you pick an action, receive a reward and then land in a new situation, drawn at random from odds that depend only on the current situation and your choice.
- AI agentAI agent, n.: An AI system that does not just answer but acts: given a goal, it decides on steps, uses tools such as a browser or code editor, checks the results and keeps going until the job is done.
- credit assignment problemcredit assignment problem, n.: The puzzle of figuring out which of many earlier decisions, or which parts of a system, deserve the credit or blame for a final outcome — like working out which moves won a chess game.
- multi-agent systemmulti-agent system, n.: A setup in which several AI agents, each with its own role or point of view, work together — or compete — on a problem, passing messages and results among themselves.
Sources
- Littman, M. L. (1994). Markov games as a framework for multi-agent reinforcement learning. Proceedings of ICML 1994.
- Tan, M. (1993). Multi-Agent Reinforcement Learning: Independent vs. Cooperative Agents. Proceedings of ICML 1993.
- Shapley, L. S. (1953). Stochastic Games. Proceedings of the National Academy of Sciences 39(10), 1095–1100.
Mentioned in
environmentenvironment, n.: Everything an AI agent interacts with but does not control — a game, a simulator, a website, the physical world. It answers each of the agent's actions with a new situation and, in reinforcement learning, a score., game theorygame theory, n.: The mathematics of strategic decisions, where each player's best move depends on what the others do. It is used to study competition and cooperation among people, firms, nations and AI agents., multi-agent systemmulti-agent system, n.: A setup in which several AI agents, each with its own role or point of view, work together — or compete — on a problem, passing messages and results among themselves., self-playself-play, n.: Training an AI by having it play against copies of itself, so it always has an opponent at its own level and can keep improving without human examples., agentic reinforcement learningagentic reinforcement learning, n.: Training an AI assistant by rewarding success on long, multi-step jobs where it must use tools, browse, run code and react to what happens, rather than answer in one go., AlphaStarAlphaStar, prop. n.: A DeepMind AI that learned to play the real-time strategy game StarCraft II well enough to reach its highest public rank, Grandmaster, in 2019., counterfactual regret minimizationcounterfactual regret minimization, n.: A method for working out near-unbeatable strategies in games with hidden information, like poker, by repeatedly playing against itself and shifting each decision toward the moves it most regrets not making., Nash equilibriumNash equilibrium, n.: A stable standoff in a game: each player's choice is already their best answer to what the others are doing, so nobody gains by changing course alone — though the result need not be good for anyone., OpenAI FiveOpenAI Five, prop. n.: A team of five AI players built by OpenAI that learned the team video game Dota 2 by playing against itself, and in April 2019 beat the reigning world-champion human team., swarm roboticsswarm robotics, n.: Coordinating large groups of simple robots that follow local rules and talk only to their neighbors, so useful group behavior emerges without a central controller, as in ant colonies or bird flocks.