Abstract:Cellular-connected unmanned aerial vehicle (UAV) is a promising technology to unlock the full potential of UAVs in the future by reusing the cellular base stations (BSs) to enable their air-ground communications. However, how to achieve ubiquitous three-dimensional (3D) communication coverage for the UAVs in the sky is a new challenge. In this paper, we tackle this challenge by a new coverage-aware navigation approach, which exploits the UAV's controllable mobility to design its navigation/trajectory to avoid the cellular BSs' coverage holes while accomplishing their missions. To this end, we formulate an UAV trajectory optimization problem to minimize the weighted sum of its mission completion time and expected communication outage duration, which, however, cannot be solved by the standard optimization techniques due to the lack of an accurate and tractable end-to-end communication model in practice. To overcome this difficulty, we propose a new solution approach based on the technique of deep reinforcement learning (DRL). Specifically, by leveraging the state-of-the-art dueling double deep Q network (dueling DDQN) with multi-step learning, we first propose a UAV navigation algorithm based on direct RL, where the signal measurement at the UAV is used to directly train the action-value function of the navigation policy. To further improve the performance, we propose a new framework called simultaneous navigation and radio mapping (SNARM), where the UAV's signal measurement is used not only for training the DQN directly, but also to create a radio map that is able to predict the outage probabilities at all locations in the area of interest. This enables the generation of simulated UAV trajectories and predicting their expected returns, which are then used to further train the DQN via Dyna technique, thus great-y improving the learning efficiency.

Reinforcement Learning for a Cellular Internet of UAVs: Protocol Design, Trajectory Control, and Resource Management

Reinforcement Learning for Decentralized Trajectory Design in Cellular UAV Networks with Sense-and-Send Protocol.

Cooperative Internet of UAVs: Distributed Trajectory Design by Multi-Agent Deep Reinforcement Learning.

Distributed Trajectory Design for Cooperative Internet of UAVs Using Deep Reinforcement Learning

Cellular UAV-to-Device Communications: Trajectory Design and Mode Selection by Multi-agent Deep Reinforcement Learning

Deep Reinforcement Learning for Joint Trajectory Planning, Transmission Scheduling, and Access Control in UAV-Assisted Wireless Sensor Networks

Robust and Decentralized Reinforcement Learning for UAV Path Planning in IoT Networks

Simultaneous Navigation and Radio Mapping for Cellular-Connected UAV With Deep Reinforcement Learning

Model-aided Deep Reinforcement Learning for Sample-efficient UAV Trajectory Design in IoT Networks

Radio Resource Management for Cellular-Connected UAV: A Learning Approach

Deep Reinforcement Learning for Interference Management in UAV-based 3D Networks: Potentials and Challenges

Research on the UAV-aided Data Collection and Trajectory Design Based on the Deep Reinforcement Learning

Trajectory Design for UAV-Based Internet of Things Data Collection: A Deep Reinforcement Learning Approach

Deep Reinforcement Learning for Unmanned Aerial Vehicle-Assisted Vehicular Networks

Deep Reinforcement Learning-Based 3D Trajectory Planning for Cellular Connected UAV

UAV Path Planning for Wireless Data Harvesting: A Deep Reinforcement Learning Approach

Three-dimensional deep reinforcement learning for trajectory and resource optimization in UAV communication systems

Deep Reinforcement Learning Based 3D UAV Trajectory Design and Frequency Band Allocation.

UAV-to-Device Underlay Communications: Age of Information Minimization by Multi-agent Deep Reinforcement Learning

Cellular-Connected UAVs over 5G: Deep Reinforcement Learning for Interference Management

Deep Reinforcement Learning for UAV Trajectory Design Considering Mobile Ground Users