Abstract:Multiple unmanned aerial vehicles (Multi-UAV) systems have recently demonstrated significant advantages in some real-world scenarios, but the limited communication range of UAVs poses great challenges to multi-UAV collaborative decision-making. By constructing the multi-UAV cooperation problem as a multi-agent system (MAS), the cooperative decision-making among UAVs can be realized by using multi-agent reinforcement learning (MARL). Following this paradigm, this work focuses on developing partially observable MARL models that capture important information from local observations in order to select effective actions. Previous related studies employ either probability distributions or weighted mean field to update the average actions of neighborhood agents. However, they do not fully consider the feature information of surrounding neighbors, resulting in a local optimum often. In this paper, we propose a novel partially multi-agent reinforcement learning algorithm to remedy this flaw, which is based on graph attention network and partially observable mean field and is named as the GPMF algorithm for short. GPMF uses a graph attention module and a mean field module to describe how an agent is influenced by the actions of other agents at each time step. The graph attention module consists of a graph attention encoder and a differentiable attention mechanism, outputting a dynamic graph to represent the effectiveness of neighborhood agents against central agents. The mean field module approximates the effect of a neighborhood agent on a central agent as the average effect of effective neighborhood agents. Aiming at the typical task scenario of large-scale multi-UAV cooperative roundup, the proposed algorithm is evaluated based on the MAgent framework. Experimental results show that GPMF outperforms baselines including state-of-the-art partially observable mean field reinforcement learning algorithms, providing technical support for large-scale multi-UAV coordination and confrontation tasks in communication-constrained environments.

Multi-Agent Partial Observable Safe Reinforcement Learning for Counter Uncrewed Aerial Systems

High-Sample-Efficient Multiagent Reinforcement Learning for Navigation and Collision Avoidance of UAV Swarms in Multitask Environments

UAV Cooperative Air Combat Maneuvering Confrontation Based on Multi-agent Reinforcement Learning

Multi-UAV Cooperative Search in Multi-Layered Aerial Computing Networks: A Multi-Agent Deep Reinforcement Learning Approach

Cooperative Multi-UAV Dynamic Anti-Jamming Scheme with Deep Reinforcement Learning.

Autonomous and cooperative control of UAV cluster with multi-agent reinforcement learning

An evolutionary multi-agent reinforcement learning algorithm for multi-UAV air combat

Mean-Field Multi-Agent Reinforcement Learning for UAV Assisted Secure Data Dissemination.

Partially Observable Mean Field Multi-Agent Reinforcement Learning Based on Graph Attention Network for UAV Swarms

Cooperative Pursuit of Unauthorized UAVs in Urban Airspace Via Multi-agent Reinforcement Learning

Matching combined multi-agent reinforcement learning for uav secure data dissemination

Multi-Target Pursuit by a Decentralized Heterogeneous UAV Swarm using Deep Multi-Agent Reinforcement Learning

Safe Multi-Agent Reinforcement Learning for Behavior-Based Cooperative Navigation

Multi-UAV Escape Target Search: A Multi-Agent Reinforcement Learning Method

Multi-Agent Reinforcement Learning for Maritime Operational Technology Cyber Security

UAV-Enabled Secure Communications by Multi-Agent Deep Reinforcement Learning

Maritime Search and Rescue Leveraging Heterogeneous Units: A Multi-Agent Reinforcement Learning Approach.

Robust Multi-Agent Reinforcement Learning Method Based on Adversarial Domain Randomization for Real-World Dual-UAV Cooperation

Strategic Conflict Management using Recurrent Multi-agent Reinforcement Learning for Urban Air Mobility Operations Considering Uncertainties

Integrating unmanned and manned UAVs data network based on combined Bayesian belief network and multi-objective reinforcement learning algorithm

Multi-Agent Deep Reinforcement Learning for Secure UAV Communications