Abstract:Recent advancements in reinforcement learning have witnessed remarkable achievements by intelligent agents ranging from game-playing to industrial applications. Of particular interest is the area of multi-agent reinforcement learning (MARL), which holds significant potential for real-world scenarios. However, typical MARL methods are limited in their ability to handle tens of agents, leaving scenarios with up to hundreds or even thousands of agents almost unexplored. The scaling up of the number of agents presents two primary challenges: (1) agent-agent interactions are crucial in multi-agent systems while the number of interactions grows quadratically with the number of agents, resulting in substantial computational complexity and difficulty in strategies-learning; (2) the strengths of interactions among agents exhibit variations both across agents and over time, making it difficult to precisely model such interactions. In this paper, we propose a novel approach named Graph Attention Mean Field (GAT-MF). By converting agent-agent interactions into interactions between each agent and a weighted mean field, we achieve a substantial reduction in computational complexity. The proposed method offers a precise modeling of interaction dynamics with mathematical proofs of its correctness. Additionally, we design a graph attention mechanism to automatically capture the diverse and time-varying strengths of interactions, ensuring an accurate representation of agent interactions. Through extensive experimentation conducted in both manual and real-world scenarios involving over 3000 agents, we validate the efficacy of our method. The results demonstrate that our method outperforms the best baseline method with a remarkable improvement of 42.7%. Furthermore, our method saves 86.4% training time and 19.2% GPU memory compared to the best baseline method.

AVD-Net: Attention Value Decomposition Network for Deep Multi-Agent Reinforcement Learning

VAAC: V-value Attention Actor-Critic for Cooperative Multi-agent Reinforcement Learning.

Boosting Value Decomposition Via Unit-Wise Attentive State Representation for Cooperative Multi-Agent Reinforcement Learning

Adaptive Value Decomposition with Greedy Marginal Contribution Computation for Cooperative Multi-Agent Reinforcement Learning

Attentive Relational State Representation in Decentralized Multiagent Reinforcement Learning.

Value-Decomposition Networks For Cooperative Multi-Agent Learning

Weighted Mean-Field Multi-Agent Reinforcement Learning via Reward Attribution Decomposition

Modeling the Interaction Between Agents in Cooperative Multi-Agent Reinforcement Learning

SVDE: Scalable Value-Decomposition Exploration for Cooperative Multi-Agent Reinforcement Learning

Q-value Path Decomposition for Deep Multiagent Reinforcement Learning

Maximum Correntropy Value Decomposition for Multi-agent Deep Reinforcemen Learning

Multi-Agent Concentrative Coordination with Decentralized Task Representation

Learning Attentional Communication with a Common Network for Multiagent Reinforcement Learning.

Attentional Factorized Q-Learning for Many-Agent Learning.

Deconfounded Value Decomposition for Multi-Agent Reinforcement Learning

STAS: Spatial-Temporal Return Decomposition for Multi-agent Reinforcement Learning.

Inducing Cooperation via Team Regret Minimization based Multi-Agent Deep Reinforcement Learning

Stochastic Graph Neural Network-Based Value Decomposition for Multi-Agent Reinforcement Learning in Urban Traffic Control.

Multi-View Deep Attention Network for Reinforcement Learning (student Abstract).

GAT-MF: Graph Attention Mean Field for Very Large Scale Multi-Agent Reinforcement Learning