Abstract:Recent advancements in reinforcement learning have witnessed remarkable achievements by intelligent agents ranging from game-playing to industrial applications. Of particular interest is the area of multi-agent reinforcement learning (MARL), which holds significant potential for real-world scenarios. However, typical MARL methods are limited in their ability to handle tens of agents, leaving scenarios with up to hundreds or even thousands of agents almost unexplored. The scaling up of the number of agents presents two primary challenges: (1) agent-agent interactions are crucial in multi-agent systems while the number of interactions grows quadratically with the number of agents, resulting in substantial computational complexity and difficulty in strategies-learning; (2) the strengths of interactions among agents exhibit variations both across agents and over time, making it difficult to precisely model such interactions. In this paper, we propose a novel approach named Graph Attention Mean Field (GAT-MF). By converting agent-agent interactions into interactions between each agent and a weighted mean field, we achieve a substantial reduction in computational complexity. The proposed method offers a precise modeling of interaction dynamics with mathematical proofs of its correctness. Additionally, we design a graph attention mechanism to automatically capture the diverse and time-varying strengths of interactions, ensuring an accurate representation of agent interactions. Through extensive experimentation conducted in both manual and real-world scenarios involving over 3000 agents, we validate the efficacy of our method. The results demonstrate that our method outperforms the best baseline method with a remarkable improvement of 42.7%. Furthermore, our method saves 86.4% training time and 19.2% GPU memory compared to the best baseline method.

Characterizing and Optimizing the End-to-End Performance of Multi-Agent Reinforcement Learning Systems

Towards Efficient Multi-Agent Learning Systems

Scalability Bottlenecks in Multi-Agent Reinforcement Learning Systems

Characterizing Speed Performance of Multi-Agent Reinforcement Learning

Multiagent Reinforcement Learning for Strictly Constrained Tasks Based on Reward Recorder

S2rl

Breaking the mold: The challenge of large scale MARL specialization

Efficient Multi-agent Reinforcement Learning by Planning

PPS-QMIX: Periodically Parameter Sharing for Accelerating Convergence of Multi-Agent Reinforcement Learning

MARL-LNS: Cooperative Multi-agent Reinforcement Learning via Large Neighborhoods Search

Adaptive Learning Rates for Multi-Agent Reinforcement Learning

Revisiting Some Common Practices in Cooperative Multi-Agent Reinforcement Learning

GAT-MF: Graph Attention Mean Field for Very Large Scale Multi-Agent Reinforcement Learning

Sample-Efficient Multi-Agent RL: an Optimization Perspective.

Towards a Standardised Performance Evaluation Protocol for Cooperative MARL

Learning from Good Trajectories in Offline Multi-Agent Reinforcement Learning

Taming Multi-Agent Reinforcement Learning with Estimator Variance Reduction

Multiexperience-Assisted Efficient Multiagent Reinforcement Learning

Scalable Multi-Agent Reinforcement Learning for Residential Load Scheduling under Data Governance

B2MAPO: A Batch-by-Batch Multi-Agent Policy Optimization to Balance Performance and Efficiency

Offline Multi-Agent Reinforcement Learning via In-Sample Sequential Policy Optimization