Abstract:Finding the optimal game strategy is a difficult problem in unmanned aerial vehicle (UAV) swarm confrontation. As an effective solution to the sequential decision-making problem, multi-agent reinforcement learning (MARL) provides a promising way to realize intelligent countermeasures. However, there are two challenges in applying MARL to large-scale UAV swarm confrontation: i) the curse of dimensionality caused by the excessive scale of UAV clusters and ii) the generalization problem caused by the dynamically changing UAV cluster size. To address these problems, we propose a novel MARL paradigm, called Weighted Mean Field Reinforcement Learning , where the pairwise communication between any UAV and its neighbors is modeled as that between a central UAV and the virtual UAV, which is abstracted from the weighted mean effect of neighboring UAVs. This approach reduces the multi-agent problem to a two-agent problem, which can reduce the input dimension of the agent and adapt to the changing cluster size. The communication content between UAVs includes actions and local observations. Actions can enhance the cooperation between UAVs and alleviate the non-stationarity of the environment, while local observations can expand the perception range of the central UAV so that it can obtain more useful information about the environment. The attention mechanism is leveraged to enable UAVs to select more valuable information flexibly, making our method more scalable than other algorithms. Combining this paradigm with double Q-learning and actor-critic algorithms, we propose weighted mean field Q-learning (WMFQ) and weighted mean field actor-critic (WMFAC) algorithms. Experiments on our constructed UAV swarm confrontation environment verify the effectiveness and scalability of our algorithms.

Programming and Deployment of Autonomous Swarms using Multi-Agent Reinforcement Learning

High-Sample-Efficient Multiagent Reinforcement Learning for Navigation and Collision Avoidance of UAV Swarms in Multitask Environments

Multi-Agent Reinforcement Learning Based UAV Swarm Communications Against Jamming

Collaborative Decision-making in Heterogeneous UAV Swarms Based on Multi-agent Deep Reinforcement Learning

Distributed UAV Swarm for Device-Free Integrated Sensing and Communication Relying on Multi-Agent Reinforcement Learning

Decentralized Control of Quadrotor Swarms with End-to-end Deep Reinforcement Learning

UAV Swarm Air Combat Maneuver Decision-Making Method Based on Multi-Agent Reinforcement Learning and Transferring

Control and Coordination of a SWARM of Unmanned Surface Vehicles using Deep Reinforcement Learning in ROS

UAV Swarm Confrontation Using Hierarchical Multiagent Reinforcement Learning

PASCAL: PopulAtion-Specific Curriculum-based MADRL for collision-free flocking with large-scale fixed-wing UAV swarms

UAV Swarm Cooperative Target Search: A Multi-Agent Reinforcement Learning Approach

Scalable Task-Driven Robotic Swarm Control via Collision Avoidance and Learning Mean-Field Control

Autonomous Swarm Robot Coordination via Mean-Field Control Embedding Multi-Agent Reinforcement Learning

Application of Deep Reinforcement Learning to UAV Swarming for Ground Surveillance

Task Assignment of UAV Swarms Based on Deep Reinforcement Learning

A Local Information Aggregation based Multi-Agent Reinforcement Learning for Robot Swarm Dynamic Task Allocation

Collision-Avoiding Flocking With Multiple Fixed-Wing UAVs in Obstacle-Cluttered Environments: A Task-Specific Curriculum- Based MADRL Approach

HA-MARL: Heuristic and APF Assisted Multi-Agent Reinforcement Learning for Wireless Data Sharing in AUV Swarms

Oracle-Guided Deep Reinforcement Learning for Large-Scale Multi-UAVs Flocking and Navigation.

Autonomous and cooperative control of UAV cluster with multi-agent reinforcement learning

Weighted Mean Field Reinforcement Learning for Large-Scale UAV Swarm Confrontation