Abstract:Compared with terrestrial networks, unmanned aerial vehicles (UAVs) have the characteristics of flexible deployment and strong adaptability, which are an important supplement to intelligent transportation systems (ITS). In this paper, we focus on the multi-UAV network area coverage problem (ACP) which require intelligent UAVs long-term trajectory decisions in the complex and scalable network environment. Multi-agent deep reinforcement learning (DRL) has recently emerged as an effective tool for solving long-term decisions problems. However, since the input dimension of multi-layer perceptron (MLP)-based deep neural network (DNN) is fixed, it is difficult for standard DNN to adapt to a variable number of UAVs and network users. Therefore, we combine Transformer with DRL to meet the scalability of the network and propose a Transformer-based deep multi-agent reinforcement learning (T-MARL) algorithm. Transformer can adapt to variable input dimensions and extract important information from complex network states by attention module. In our research, we find that random initialization of Transformer may cause DRL training failure, so we propose a baseline-assisted pre-training scheme. This scheme can quickly provide an initial policy model for UAVs based on imitation learning, and use the temporal-difference(1) algorithm to initialize policy evaluation network. Finally, based on parameter sharing, T-MARL is applicable to any standard DRL algorithm and supports expansion on networks of different sizes. Experimental results show that T-MARL can make UAVs have cooperative behaviors and perform outstandingly on ACP.

Ensuring Threshold AoI for UAV-Assisted Mobile Crowdsensing by Multi-Agent Deep Reinforcement Learning with Transformer

AoI-minimal UAV Crowdsensing by Model-based Graph Convolutional Reinforcement Learning

High-Performance UAV Crowdsensing: A Deep Reinforcement Learning Approach

Transformer-Based Reinforcement Learning for Scalable Multi-UAV Area Coverage

Deep Reinforcement Learning-Based Unmanned Aerial Vehicle Mobile Crowdsensing with Landing Constraints

Delay-Sensitive Energy-Efficient UAV Crowdsensing by Deep Reinforcement Learning

Work Together to Keep Fresh: Hierarchical Learning for UAVs-assisted Data Time-Sensitive IoT

QoI-Aware Mobile Crowdsensing for Metaverse by Multi-Agent Deep Reinforcement Learning

Cooperative Data Sensing and Computation Offloading in UAV-Assisted Crowdsensing with Multi-Agent Deep Reinforcement Learning

Cellular UAV-to-Device Communications: Trajectory Design and Mode Selection by Multi-agent Deep Reinforcement Learning

Maximizing UAV Coverage in Maritime Wireless Networks: A Multiagent Reinforcement Learning Approach

Exploring Both Individuality and Cooperation for Air-Ground Spatial Crowdsourcing by Multi-Agent Deep Reinforcement Learning.

Mobile Crowdsensing for Data Freshness: A Deep Reinforcement Learning Approach.

Multi-Task-Oriented UAV Crowd Sensing with Charging Budget Constraint

Learning-Based Energy-Efficient Data Collection by Unmanned Vehicles in Smart Cities

Multi-Task-Oriented Vehicular Crowdsensing: A Deep Learning Approach

Trajectory Design for Multi-UAV-Assisted Data Collection: A Multi-agent Deep Reinforcement Learning Approach

Energy-Efficient UAV Crowdsensing with Multiple Charging Stations by Deep Learning

Joint Optimization of Multi-UAV Deployment and User Association Via Deep Reinforcement Learning for Long-Term Communication Coverage

When Crowdsourcing Meets Unmanned Vehicles: Toward Cost-Effective Collaborative Urban Sensing Via Deep Reinforcement Learning

Deep RL-based Trajectory Planning for AoI Minimization in UAV-assisted IoT