Abstract:Value function factorization via centralized training and decentralized execution is promising for solving cooperative multi-agent reinforcement tasks. One of the approaches in this area, QMIX, has become state-of-the-art and achieved the best performance on the StarCraft II micromanagement benchmark. However, the monotonic-mixing of per agent estimates in QMIX is known to restrict the joint action Q-values it can represent, as well as the insufficient global state information for single agent value function estimation, often resulting in suboptimality. To this end, we present LSF-SAC, a novel framework that features a variational inference-based information-sharing mechanism as extra state information to assist individual agents in the value function factorization. We demonstrate that such latent individual state information sharing can significantly expand the power of value function factorization, while fully decentralized execution can still be maintained in LSF-SAC through a soft-actor-critic design. We evaluate LSF-SAC on the StarCraft II micromanagement challenge and demonstrate that it outperforms several state-of-the-art methods in challenging collaborative tasks. We further set extensive ablation studies for locating the key factors accounting for its performance improvements. We believe that this new insight can lead to new local value estimation methods and variational deep learning algorithms. A demo video and code of implementation can be found at <a class="link-external link-https" href="https://sites.google.com/view/sacmm" rel="external noopener nofollow">this https URL</a>.

Credit-of-Q-value for Multi-Agent Reinforcement Learning

Shapley Q-Value: A Local Reward Approach to Solve Global Reward Games

Towards Understanding Cooperative Multi-Agent Q-Learning with Value Factorization.

Priority over Quantity: A Self-Incentive Credit Assignment Scheme for Cooperative Multiagent Reinforcement Learning

ConcaveQ: Non-Monotonic Value Function Factorization Via Concave Representations in Deep Multi-Agent Reinforcement Learning

Qrelation: an Agent Relation-Based Approach for Multi-Agent Reinforcement Learning Value Function Factorization.

Qatten: A General Framework for Cooperative Multiagent Reinforcement Learning

QFree: A Universal Value Function Factorization for Multi-Agent Reinforcement Learning

Transform Networks for Cooperative Multi-Agent Deep Reinforcement Learning

ResQ: A Residual Q Function-based Approach for Multi-Agent Reinforcement Learning Value Factorization

DVF:Multi-agent Q-learning with difference value factorization

Q-value Path Decomposition for Deep Multiagent Reinforcement Learning

Multi-Agent Q-Value Mixing Network with Covariance Matrix Adaptation Strategy for the Voltage Regulation Problem

Value Functions Factorization with Latent State Information Sharing in Decentralized Multi-Agent Policy Gradients

Multiagent Q-learning with Sub-Team Coordination.

Learning Nearly Decomposable Value Functions Via Communication Minimization

SQIX: QMIX Algorithm Activated by General Softmax Operator for Cooperative Multiagent Reinforcement Learning

Reducing overestimation in value mixing for cooperative deep multi-agent reinforcement learning

POWQMIX: Weighted Value Factorization with Potentially Optimal Joint Actions Recognition for Cooperative Multi-Agent Reinforcement Learning

Learning Multi-Agent Cooperation via Considering Actions of Teammates

QGNN: Value Function Factorisation with Graph Neural Networks