Abstract:Unmanned aerial vehicle (UAV) swarms have found extensive applications in various fields,playing a crucial role in cluster collaboration. These swarms involve multiple UAVs that work together to achieve common objectives. A key challenging task in swarm operations is collision-free formation control of UAVs. To solve this problem,applying deep reinforcement learning methods has received significant attention,but their application on autonomous UAVs poses challenges,including dependency on global information during training,difficulties in sampling,and excessive resource utilization. To overcome these challenges,in this work,a novel approach based on multi-agent deep reinforcement learning (MARL) is proposed for collision-free formation control of UAV swarms. MARL allows each UAV to interact with a dynamic environment that includes other UAVs,enabling collaborative decision-making and adaptive behavior. We focus on leveraging local information to establish a state space for individual UAVs. To train the policy network,we employ the multi-agent proximal policy optimization (MAPPO) algorithm,allowing robust learning and policy optimization in a multi-agent setting. Also,we address the issues of sampling difficulties and resource constraints by utilizing digital twin technology,serving as a bridge between physical entities and virtual models,which offers a novel approach to the intelligent collaborative control of drone swarms. By establishing models in virtual space,digital twin technology enables the simulation of real-world spaces for pre-training the reinforcement learning algorithm by generating synthetic experiences. We construct multiple digital twin environments to facilitate interactive sampling and pre-train the swarm with basic task capabilities. Then,we supplement the training using real-world data collected in actual environments,enhancing the ability of the swarm to perform optimally in real-world scenarios. To evaluate the effectiveness of our approach,we compare the performance of the two-stage training architecture with other policy algorithms. To validate the sample efficiency of the on-policy algorithm MAPPO,we conducted a comparative analysis with other policy algorithms,particularly off-policy algorithms. The results reveal the superior sample efficiency and stability of MAPPO in addressing the challenges of collision-free formation control. Finally,we conduct a real-flight validation test to validate the practicality and reliability of the strategy model derived from the digital twin environments. Overall,this work demonstrates the effectiveness of our proposed approach in enabling UAV swarms to navigate complex environments and achieve collision-free formation control.

Coordinating Team Tactics for Swarm-Versus-Swarm Adversarial Games

An Effective and Scalable Approach for Swarm-on-Swarm Air Combat Decision

Learning of Coordination Policies for Robotic Swarms

UAV Swarm Air Combat Maneuver Decision-Making Method Based on Multi-Agent Reinforcement Learning and Transferring

High-Sample-Efficient Multiagent Reinforcement Learning for Navigation and Collision Avoidance of UAV Swarms in Multitask Environments

Autonomous maneuver strategy of swarm air combat based on DDPG

Modeling and Control of Large-Scale Adversarial Swarm Engagements

UAV Swarm Confrontation Based on Multi-agent Deep Reinforcement Learning

Hawk-Pigeon Game Tactics for Unmanned Aerial Vehicle Swarm Target Defense

SwarmPlay: Interactive Tic-tac-toe Board Game with Swarm of Nano-UAVs driven by Reinforcement Learning

King of the Hill: C2 for Next Generation Swarm Warfare

UAV Swarm Confrontation Using Hierarchical Multiagent Reinforcement Learning

Weighted Mean Field Reinforcement Learning for Large-Scale UAV Swarm Confrontation

A Bio-Inspired Decision-Making Method of UAV Swarm for Attack-Defense Confrontation via Multi-Agent Reinforcement Learning

Application of Deep Reinforcement Learning to Defense and Intrusion Strategies Using Unmanned Aerial Vehicles in a Versus Game

Hierarchical Reinforcement Learning for Swarm Confrontation with High Uncertainty

Towards Physically Talented Aerial Robots with Tactically Smart Swarm Behavior thereof: An Efficient Co-design Approach

Programming and Deployment of Autonomous Swarms using Multi-Agent Reinforcement Learning

A Scalable Reinforcement Learning Approach for Attack Allocation in Swarm to Swarm Engagement Problems

Digital Twin-Based Obstacle Avoidance Method for Unmanned Aerial Vehicle Formation Control Using Deep Reinforcement Learning

Scalable Task-Driven Robotic Swarm Control via Collision Avoidance and Learning Mean-Field Control