Abstract:Traditional multi-agent reinforcement learning algorithms are difficultly applied in a large-scale multi-agent environment. The introduction of mean field theory has enhanced the scalability of multi-agent reinforcement learning in recent years. This paper considers partially observable multi-agent reinforcement learning (MARL), where each agent can only observe other agents within a fixed range. This partial observability affects the agent's ability to assess the quality of the actions of surrounding agents. This paper focuses on developing a method to capture more effective information from local observations in order to select more effective actions. Previous work in this field employs probability distributions or weighted mean field to update the average actions of neighborhood agents, but it does not fully consider the feature information of surrounding neighbors and leads to a local optimum. In this paper, we propose a novel multi-agent reinforcement learning algorithm, Partially Observable Mean Field Multi-Agent Reinforcement Learning based on Graph--Attention (GAMFQ) to remedy this flaw. GAMFQ uses a graph attention module and a mean field module to describe how an agent is influenced by the actions of other agents at each time step. This graph attention module consists of a graph attention encoder and a differentiable attention mechanism, and this mechanism outputs a dynamic graph to represent the effectiveness of neighborhood agents against central agents. The mean--field module approximates the effect of a neighborhood agent on a central agent as the average effect of effective neighborhood agents. We evaluate GAMFQ on three challenging tasks in the MAgents framework. Experiments show that GAMFQ outperforms baselines including the state-of-the-art partially observable mean-field reinforcement learning algorithms.

Graph-QMIX: Addressing the Partial Observation Issues Via Graph Neural Network in Multi-Agent Reinforcement Learning

Partially Observable Mean Field Multi-Agent Reinforcement Learning Based on Graph-Attention

Regularized Softmax Deep Multi-Agent Q-Learning.

Reducing overestimation in value mixing for cooperative deep multi-agent reinforcement learning

QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning

Scene Dynamics: Counterfactual Critic Multi-Agent Training for Scene Graph Generation.

Counterfactual Critic Multi-Agent Training for Scene Graph Generation

Multi-Agent Q-Value Mixing Network with Covariance Matrix Adaptation Strategy for the Voltage Regulation Problem

QGNN: Value Function Factorisation with Graph Neural Networks

Scalable spectral representations for multi-agent reinforcement learning in network MDPs

SQIX: QMIX Algorithm Activated by General Softmax Operator for Cooperative Multiagent Reinforcement Learning

MAR2MIX: A Novel Model for Dynamic Problem in Multi-agent Reinforcement Learning.

Scalable and Transferable Reinforcement Learning for Multi-Agent Mixed Cooperative–Competitive Environments Based on Hierarchical Graph Attention

MAGNet: Multi-agent Graph Network for Deep Multi-agent Reinforcement Learning

GMIX: Graph-based spatial–temporal multi-agent reinforcement learning for dynamic electric vehicle dispatching system

Soft-HGRNs: Soft Hierarchical Graph Recurrent Networks for Multi-Agent Partially Observable Environments

CMIX: Deep Multi-agent Reinforcement Learning with Peak and Average Constraints

Graph MADDPG with RNN for multiagent cooperative environment

Multiagent Reinforcement Learning With Heterogeneous Graph Attention Network

DQMIX: A Distributional Perspective on Multi-Agent Reinforcement Learning

Mix-ME: Quality-Diversity for Multi-Agent Learning