Abstract:Finding the optimal game strategy is a difficult problem in unmanned aerial vehicle (UAV) swarm confrontation. As an effective solution to the sequential decision-making problem, multi-agent reinforcement learning (MARL) provides a promising way to realize intelligent countermeasures. However, there are two challenges in applying MARL to large-scale UAV swarm confrontation: i) the curse of dimensionality caused by the excessive scale of UAV clusters and ii) the generalization problem caused by the dynamically changing UAV cluster size. To address these problems, we propose a novel MARL paradigm, called Weighted Mean Field Reinforcement Learning , where the pairwise communication between any UAV and its neighbors is modeled as that between a central UAV and the virtual UAV, which is abstracted from the weighted mean effect of neighboring UAVs. This approach reduces the multi-agent problem to a two-agent problem, which can reduce the input dimension of the agent and adapt to the changing cluster size. The communication content between UAVs includes actions and local observations. Actions can enhance the cooperation between UAVs and alleviate the non-stationarity of the environment, while local observations can expand the perception range of the central UAV so that it can obtain more useful information about the environment. The attention mechanism is leveraged to enable UAVs to select more valuable information flexibly, making our method more scalable than other algorithms. Combining this paradigm with double Q-learning and actor-critic algorithms, we propose weighted mean field Q-learning (WMFQ) and weighted mean field actor-critic (WMFAC) algorithms. Experiments on our constructed UAV swarm confrontation environment verify the effectiveness and scalability of our algorithms.

An evolutionary multi-agent reinforcement learning algorithm for multi-UAV air combat

UAV Cooperative Air Combat Maneuvering Confrontation Based on Multi-agent Reinforcement Learning

Evolutionary reinforcement learning algorithm for large-scale multi-agent cooperation and confrontation applications

Autonomous and cooperative control of UAV cluster with multi-agent reinforcement learning

Weighted Mean Field Reinforcement Learning for Large-Scale UAV Swarm Confrontation

Collaborative Decision-Making Method for Multi-UAV Based on Multiagent Reinforcement Learning

UAV cooperative air combat maneuver decision based on multi-agent reinforcement learning

UAV Swarm Confrontation Using Hierarchical Multiagent Reinforcement Learning

Enhancing multi-UAV air combat decision making via hierarchical reinforcement learning

Multi-Agent Partial Observable Safe Reinforcement Learning for Counter Uncrewed Aerial Systems

Hierarchical Multi-Agent Reinforcement Learning for Air Combat Maneuvering

Game of Drones: Intelligent Online Decision Making of Multi-UAV Confrontation

Multi-UAV Escape Target Search: A Multi-Agent Reinforcement Learning Method

General multi-agent reinforcement learning integrating adaptive manoeuvre strategy for real-time multi-aircraft conflict resolution

Underwater Multi-agent Cooperative Formation Hunting Based on Deep Reinforcement Learning

An Improved PPO for Multiple Unmanned Aerial Vehicles

A Bio-Inspired Decision-Making Method of UAV Swarm for Attack-Defense Confrontation via Multi-Agent Reinforcement Learning

Multi-UAV Pursuit-Evasion with Online Planning in Unknown Environments by Deep Reinforcement Learning

Dense Multi-Agent Reinforcement Learning Aided Multi-UAV Information Coverage for Vehicular Networks

Multi-Agent Reinforcement Learning Aided Intelligent UAV Swarm for Target Tracking

Multi-UAV Collaborative Search and Strike based on Reinforcement Learning