QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning

Tabish Rashid,Mikayel Samvelyan,Christian Schroeder de Witt,Gregory Farquhar,Jakob Foerster,Shimon Whiteson

DOI: https://doi.org/10.48550/arXiv.1803.11485

2018-06-07

Abstract:In many real-world settings, a team of agents must coordinate their behaviour while acting in a decentralised way. At the same time, it is often possible to train the agents in a centralised fashion in a simulated or laboratory setting, where global state information is available and communication constraints are lifted. Learning joint action-values conditioned on extra state information is an attractive way to exploit centralised learning, but the best strategy for then extracting decentralised policies is unclear. Our solution is QMIX, a novel value-based method that can train decentralised policies in a centralised end-to-end fashion. QMIX employs a network that estimates joint action-values as a complex non-linear combination of per-agent values that condition only on local observations. We structurally enforce that the joint-action value is monotonic in the per-agent values, which allows tractable maximisation of the joint action-value in off-policy learning, and guarantees consistency between the centralised and decentralised policies. We evaluate QMIX on a challenging set of StarCraft II micromanagement tasks, and show that QMIX significantly outperforms existing value-based multi-agent reinforcement learning methods.

Machine Learning,Multiagent Systems

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to learn the joint action - value function in the centralized training process in multi - agent reinforcement learning, and at the same time be able to extract effective decentralized policies. Specifically, the paper proposes the QMIX method, aiming to overcome the limitations of existing methods when dealing with multi - agent collaborative tasks. For example, independent Q - learning (IQL) cannot represent the interactions between agents, and fully centralized Q - learning is difficult to scale to a large number of agents. QMIX combines the value functions of each agent by introducing a mixing network to form a global joint action - value function, and ensures that the local optimal solution of each agent can be derived from the global optimal solution through monotonicity constraints, thus achieving the consistency between centralized training and decentralized execution. This method can not only handle complex multi - agent environments, but also utilize additional state information during the training process to improve learning efficiency and policy performance.

QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning

POWQMIX: Weighted Value Factorization with Potentially Optimal Joint Actions Recognition for Cooperative Multi-Agent Reinforcement Learning

Value Functions Factorization with Latent State Information Sharing in Decentralized Multi-Agent Policy Gradients

SQIX: QMIX Algorithm Activated by General Softmax Operator for Cooperative Multiagent Reinforcement Learning

Reducing overestimation in value mixing for cooperative deep multi-agent reinforcement learning

Soft-QMIX: Integrating Maximum Entropy For Monotonic Value Function Factorization

Value function factorization with dynamic weighting for deep multi-agent reinforcement learning

ConcaveQ: Non-Monotonic Value Function Factorization Via Concave Representations in Deep Multi-Agent Reinforcement Learning

ReMIX: Regret Minimization for Monotonic Value Function Factorization in Multiagent Reinforcement Learning

QFree: A Universal Value Function Factorization for Multi-Agent Reinforcement Learning

DQMIX: A Distributional Perspective on Multi-Agent Reinforcement Learning

Learning Multi-Agent Cooperation via Considering Actions of Teammates

On Stateful Value Factorization in Multi-Agent Reinforcement Learning

CoMIX: A Multi-agent Reinforcement Learning Training Architecture for Efficient Decentralized Coordination and Independent Decision-Making

Rethinking the Implementation Tricks and Monotonicity Constraint in Cooperative Multi-Agent Reinforcement Learning

Value-Decomposition Multi-Agent Actor-Critics

Mixed Q-Functionals: Advancing Value-Based Methods in Cooperative MARL with Continuous Action Domains

Regularized Softmax Deep Multi-Agent Q-Learning.

Learning Nearly Decomposable Value Functions Via Communication Minimization

Maximum Correntropy Value Decomposition for Multi-agent Deep Reinforcemen Learning

QTypeMix: Enhancing Multi-Agent Cooperative Strategies through Heterogeneous and Homogeneous Value Decomposition