Abstract: In cooperative multi-agent tasks, a team of agents jointly interact with an environment by taking actions, receiving a team reward and observing the next state. During the interactions, the uncertainty of environment and reward will inevitably induce stochasticity in the long-term returns and the randomness can be exacerbated with the increasing number of agents. However, most of the existing value-based multi-agent reinforcement learning (MARL) methods only model the expectations of individual Q-values and global Q-value, ignoring such randomness. Compared to the expectations of the long-term returns, it is more preferable to directly model the stochasticity by estimating the returns through distributions. With this motivation, this work proposes DQMIX, a novel value-based MARL method, from a distributional perspective. Specifically, we model each individual Q-value with a categorical distribution. To integrate these individual Q-value distributions into the global Q-value distribution, we design a distribution mixing network, based on five basic operations on the distribution. We further prove that DQMIX satisfies the \emph{Distributional-Individual-Global-Max} (DIGM) principle with respect to the expectation of distribution, which guarantees the consistency between joint and individual greedy action selections in the global Q-value and individual Q-values. To validate DQMIX, we demonstrate its ability to factorize a matrix game with stochastic rewards. Furthermore, the experimental results on a challenging set of StarCraft II micromanagement tasks show that DQMIX consistently outperforms the value-based multi-agent reinforcement learning baselines.

Value function factorization with dynamic weighting for deep multi-agent reinforcement learning

POWQMIX: Weighted Value Factorization with Potentially Optimal Joint Actions Recognition for Cooperative Multi-Agent Reinforcement Learning

QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning

Reducing overestimation in value mixing for cooperative deep multi-agent reinforcement learning

ConcaveQ: Non-Monotonic Value Function Factorization Via Concave Representations in Deep Multi-Agent Reinforcement Learning

On Stateful Value Factorization in Multi-Agent Reinforcement Learning

DVF:Multi-agent Q-learning with difference value factorization

Towards Understanding Cooperative Multi-Agent Q-Learning with Value Factorization.

Expert demonstrations guide reward decomposition for multi-agent cooperation

ResQ: A Residual Q Function-based Approach for Multi-Agent Reinforcement Learning Value Factorization

ReMIX: Regret Minimization for Monotonic Value Function Factorization in Multiagent Reinforcement Learning

Soft-QMIX: Integrating Maximum Entropy For Monotonic Value Function Factorization

Multi-agent Dueling Q-learning with Mean Field and Value Decomposition

Qatten: A General Framework for Cooperative Multiagent Reinforcement Learning

Efficient Multiagent Policy Optimization Based on Weighted Estimators in Stochastic Cooperative Environments

QFree: A Universal Value Function Factorization for Multi-Agent Reinforcement Learning

Deep Factorized Q-Learning for Large Scale Multi-Agent Learning

SQIX: QMIX Algorithm Activated by General Softmax Operator for Cooperative Multiagent Reinforcement Learning

DQMIX: A Distributional Perspective on Multi-Agent Reinforcement Learning

Maximum Correntropy Value Decomposition for Multi-agent Deep Reinforcemen Learning

Learning Nearly Decomposable Value Functions Via Communication Minimization