Partially Observable Mean Field Multi-Agent Reinforcement Learning Based on Graph-Attention

Min Yang,Guanjun Liu,Ziyuan Zhou

DOI: https://doi.org/10.3390/drones7070476

2024-09-06

Abstract:Traditional multi-agent reinforcement learning algorithms are difficultly applied in a large-scale multi-agent environment. The introduction of mean field theory has enhanced the scalability of multi-agent reinforcement learning in recent years. This paper considers partially observable multi-agent reinforcement learning (MARL), where each agent can only observe other agents within a fixed range. This partial observability affects the agent's ability to assess the quality of the actions of surrounding agents. This paper focuses on developing a method to capture more effective information from local observations in order to select more effective actions. Previous work in this field employs probability distributions or weighted mean field to update the average actions of neighborhood agents, but it does not fully consider the feature information of surrounding neighbors and leads to a local optimum. In this paper, we propose a novel multi-agent reinforcement learning algorithm, Partially Observable Mean Field Multi-Agent Reinforcement Learning based on Graph--Attention (GAMFQ) to remedy this flaw. GAMFQ uses a graph attention module and a mean field module to describe how an agent is influenced by the actions of other agents at each time step. This graph attention module consists of a graph attention encoder and a differentiable attention mechanism, and this mechanism outputs a dynamic graph to represent the effectiveness of neighborhood agents against central agents. The mean--field module approximates the effect of a neighborhood agent on a central agent as the average effect of effective neighborhood agents. We evaluate GAMFQ on three challenging tasks in the MAgents framework. Experiments show that GAMFQ outperforms baselines including the state-of-the-art partially observable mean-field reinforcement learning algorithms.

Artificial Intelligence

What problem does this paper attempt to address?

The paper attempts to address the problem of how to improve the performance and scalability of multi-agent reinforcement learning (MARL) algorithms in partially observable large-scale multi-agent environments. Specifically:

1. **Limitations of existing algorithms**: Traditional multi-agent reinforcement learning algorithms are difficult to apply to large-scale multi-agent environments, especially in partially observable situations. Existing methods fail to fully utilize neighbor feature information when dealing with local observation information, making it easy to fall into local optima.

2. **Proposed new method**: The paper proposes a partially observable mean-field multi-agent reinforcement learning algorithm based on graph attention mechanism (GAMFQ), aiming to better capture effective information in local observations through the graph attention module and mean-field module, and accordingly select more effective action strategies.

3. **Main contributions**:
- Propose the GAMFQ algorithm, which can learn decentralized agent policies in partially observable environments without requiring global information;
- Theoretically prove that the GAMFQ algorithm settings are close to Nash equilibrium;
- Experiments show that GAMFQ outperforms baseline algorithms and the latest partially observable mean-field reinforcement learning algorithms in three challenging tasks.

In summary, this paper is dedicated to developing a new multi-agent reinforcement learning framework to overcome the shortcomings of traditional algorithms under partially observable conditions, thereby enhancing the decision-making capabilities of large-scale multi-agent systems.