Abstract:Unmanned aerial vehicle (UAV) swarms have found extensive applications in various fields,playing a crucial role in cluster collaboration. These swarms involve multiple UAVs that work together to achieve common objectives. A key challenging task in swarm operations is collision-free formation control of UAVs. To solve this problem,applying deep reinforcement learning methods has received significant attention,but their application on autonomous UAVs poses challenges,including dependency on global information during training,difficulties in sampling,and excessive resource utilization. To overcome these challenges,in this work,a novel approach based on multi-agent deep reinforcement learning (MARL) is proposed for collision-free formation control of UAV swarms. MARL allows each UAV to interact with a dynamic environment that includes other UAVs,enabling collaborative decision-making and adaptive behavior. We focus on leveraging local information to establish a state space for individual UAVs. To train the policy network,we employ the multi-agent proximal policy optimization (MAPPO) algorithm,allowing robust learning and policy optimization in a multi-agent setting. Also,we address the issues of sampling difficulties and resource constraints by utilizing digital twin technology,serving as a bridge between physical entities and virtual models,which offers a novel approach to the intelligent collaborative control of drone swarms. By establishing models in virtual space,digital twin technology enables the simulation of real-world spaces for pre-training the reinforcement learning algorithm by generating synthetic experiences. We construct multiple digital twin environments to facilitate interactive sampling and pre-train the swarm with basic task capabilities. Then,we supplement the training using real-world data collected in actual environments,enhancing the ability of the swarm to perform optimally in real-world scenarios. To evaluate the effectiveness of our approach,we compare the performance of the two-stage training architecture with other policy algorithms. To validate the sample efficiency of the on-policy algorithm MAPPO,we conducted a comparative analysis with other policy algorithms,particularly off-policy algorithms. The results reveal the superior sample efficiency and stability of MAPPO in addressing the challenges of collision-free formation control. Finally,we conduct a real-flight validation test to validate the practicality and reliability of the strategy model derived from the digital twin environments. Overall,this work demonstrates the effectiveness of our proposed approach in enabling UAV swarms to navigate complex environments and achieve collision-free formation control.

Reinforcement Learning Based Two-Level Control Framework of UAV Swarm for Cooperative Persistent Surveillance in an Unknown Urban Area

Reinforcement Learning-Based Low-Altitude Path Planning for UAS Swarm in Diverse Threat Environments

Local Pursuit Strategy-Inspired Cooperative Formation Flight and Collision Avoidance for UAV Cluster

Target Detection Formulti-Uavs Via Digital Pheromones and Navigation Algorithm in Unknownenvironments

A Cooperative Search and Coverage Algorithm with Controllable Revisit and Connectivity Maintenance for Multiple Unmanned Aerial Vehicles.

Reinforcement Learning-Based Swarm Control for UAVs in Static and Dynamic Multi-Obstacle Environments

Decentralized UAV Swarm Control: A Multi-Layered Architecture for Integrated Flight Mode Management and Dynamic Target Interception

Dynamic Control Scheme of Multiswarm Persistent Surveillance in a Changing Environment

Distributed UAV Swarms for 3D Urban Area Coverage with Incomplete Information Using Event-Triggered Hierarchical Reinforcement Learning

A Formation Maintenance and Reconstruction Method of UAV Swarm Based on Distributed Control with Obstacle Avoidance

Multi-UAVs Target Tracking in Urban Environment Based on Distributed Model Predictive Control and Levy Flight-Salp Swarm Algorithm

Multi-UAV Cooperative Target Tracking Based on Swarm Intelligence

Continuous Patrolling in Uncertain Environment with the UAV Swarm.

UAV Multi-Dynamic Target Interception: A Hybrid Intelligent Method Using Deep Reinforcement Learning and Fuzzy Logic

Decentralized Learning Control for Multi-UAV Swarm Simultaneous Coverage and Tracking

SMART: Vision-Based Method of Cooperative Surveillance and Tracking by Multiple UAVs in the Urban Environment.

UAV Swarm Deployment and Trajectory for 3D Area Coverage via Reinforcement Learning

Coordinated flight control of miniature fixed-wing UAV swarms: methods and experiments

Digital Twin-Based Obstacle Avoidance Method for Unmanned Aerial Vehicle Formation Control Using Deep Reinforcement Learning

Dynamic UAV Swarm Confrontation: An Imitation Based on Mobile Adaptive Networks

Swarm Intelligence in Collision-free Formation Control for Multi-UAV Systems with 3D Obstacle Avoidance Maneuvers