Abstract:In multi-UAV networks, the downlink (DL) and uplink (UL) associations between a UAV and a user equipment (UE) is typically coupled, which restricts each UE to associate to the same UAV for both DL and UL. However, this mode may not be efficient since UAV networks can be heterogeneous (e.g., multi-tier UAV networks) and can experience high link uncertainty due to the mobility of UAVs. The introduction of full-duplex communication in a multi-UAV network further complicates the UE-UAV association. For this reason, the idea of DL-UL decoupling (DUDe) is introduced in this work, with which each UE is allowed to associate with separate UAVs for UL and DL transmissions. Besides, the UE-UAV association depends on the flight trajectory of the UAVs, which makes the DUDe design challenging. In this article, we study the joint decoupled UL-DL association and trajectory design problem for full-duplex multi-UAV networks. A joint optimization problem is formulated with the objective of maximizing the UEs’ sum-rate in both UL and DL. Since the problem is non-convex with sophisticated states and an individual UAV may not know the reward functions of other UAVs, a robust partially observable Markov decision process (POMDP) model is proposed to characterize the model uncertainty. A multi-agent deep reinforcement learning (MADRL) approach is proposed which enables each UAV to select its policy in a distributed manner. To train the actor-critic neural networks in the MADRL approach, an improved clip and count-based proximal policy optimization (PPO) algorithm is developed. In particular, a modified clip distribution is designed to deal with the hard restrictions between current and old policies, and an intrinsic reward is introduced to enhance the exploration capability. Simulation results illustrate the superiority of our proposed schemes when compared to the benchmarks. The codes are made publicly available in GitHub (https://github.com/isdai/MADRL-PPO).

Distributed Federated Deep Reinforcement Learning Based Trajectory Optimization for Air-Ground Cooperative Emergency Networks

Energy-Efficient Multi-UAVs Cooperative Trajectory Optimization for Communication Coverage: An MADRL Approach

Federated deep reinforcement learning based trajectory design for UAV-assisted networks with mobile ground devices

Trajectory Design and Access Control for Air–Ground Coordinated Communications System With Multiagent Deep Reinforcement Learning

Multi-Task and Multi-Objective Joint Resource Optimization for UAV-Assisted Air-Ground Integrated Networks under Emergency Scenarios

Multi-Agent DRL for Air-to-Ground Communication Planning in UAV-Enabled IoT Networks

Joint Neural Network for Trajectory and Communication Design in Multi-DAV Systems

Joint UAV trajectory and communication design with heterogeneous multi-agent reinforcement learning

Learning-Based Cooperative Aerial and Ground Vehicle Routing for Emergency Communications

Deployment of Unmanned Aerial Vehicles in Next-Generation Wireless Communication Network Using Multi-Agent Reinforcement Learning

Adaptive Resource Allocation for Emergency Communications with Unmanned Aerial Vehicle-Assisted Free Space Optical/Radio Frequency Relay System

Three-Dimensional Trajectory and Resource Allocation Optimization in Multi-Unmanned Aerial Vehicle Multicast System: A Multi-Agent Reinforcement Learning Method

Bayesian Optimization Enhanced Deep Reinforcement Learning for Trajectory Planning and Network Formation in Multi-UAV Networks

Trajectory and Power Optimization for Multi-UAV Enabled Emergency Wireless Communications Networks

Optimizing Drone Energy Use for Emergency Communications in Disasters via Deep Reinforcement Learning

3D UAV Trajectory Design and Frequency Band Allocation for Energy-Efficient and Fair Communication: A Deep Reinforcement Learning Approach

Dynamic Trajectory and Power Control in Ultra-Dense UAV Networks: A Mean-Field Reinforcement Learning Approach

Multi-Agent Deep Reinforcement Learning for Joint Decoupled User Association and Trajectory Design in Full-Duplex Multi-UAV Networks

Multi-UAV Hierarchical Intelligent Traffic Offloading Network Optimization Based on Deep Federated Learning

Deep Reinforcement Learning for Joint Trajectory Planning, Transmission Scheduling, and Access Control in UAV-Assisted Wireless Sensor Networks

Air-Ground Coordination Communication by Multi-Agent Deep Reinforcement Learning