Abstract:This paper proposes a novel Reinforcement Learning (RL) approach for sim-to-real policy transfer of Vertical Take-Off and Landing Unmanned Aerial Vehicle (VTOL-UAV). The proposed approach is designed for VTOL-UAV landing on offshore docking stations in maritime operations. VTOL-UAVs in maritime operations encounter limitations in their operational range, primarily stemming from constraints imposed by their battery capacity. The concept of autonomous landing on a charging platform presents an intriguing prospect for mitigating these limitations by facilitating battery charging and data transfer. However, current Deep Reinforcement Learning (DRL) methods exhibit drawbacks, including lengthy training times, and modest success rates. In this paper, we tackle these concerns comprehensively by decomposing the landing procedure into a sequence of more manageable but analogous tasks in terms of an approach phase and a landing phase. The proposed architecture utilizes a model-based control scheme for the approach phase, where the VTOL-UAV is approaching the offshore docking station. In the Landing phase, DRL agents were trained offline to learn the optimal policy to dock on the offshore station. The Joint North Sea Wave Project (JONSWAP) spectrum model has been employed to create a wave model for each episode, enhancing policy generalization for sim2real transfer. A set of DRL algorithms have been tested through numerical simulations including value-based agents and policy-based agents such as Deep \textit{Q} Networks (DQN) and Proximal Policy Optimization (PPO) respectively. The numerical experiments show that the PPO agent can learn complicated and efficient policies to land in uncertain environments, which in turn enhances the likelihood of successful sim-to-real transfer.

Deep Reinforcement Learning for Continuous Docking Control of Autonomous Underwater Vehicles: A Benchmarking Study

Learning an End-To-End Policy for AUV Control Within Just Forty Minutes Using Parallel Simulation

Underwater Docking of an Under-Actuated Autonomous Underwater Vehicle: System Design and Control Implementation

Auv 3d Docking Control Using Deep Reinforcement Learning

Continuous Control for Autonomous Underwater Vehicle Path Following Using Deep Interactive Reinforcement Learning

Neural Network Model-Based Reinforcement Learning Control for AUV 3-D Path Following

Asynchronous Localization for Underwater Acoustic Sensor Networks: A Continuous Control Deep Reinforcement Learning Approach

Reinforcement learning based robot navigation using illegal actions for autonomous docking of surface vehicles in unknown environments

Aquatic Navigation: A Challenging Benchmark for Deep Reinforcement Learning

End-to-end deep reinforcement learning for control of an autonomous underwater robot with an undulating propulsor

Deep Interactive Reinforcement Learning for Path Following of Autonomous Underwater Vehicle

Deep Reinforcement Learning for Sim-to-Real Policy Transfer of VTOL-UAVs Offshore Docking Operations

Learning Autonomous Docking Operation of Fully Actuated Autonomous Surface Vessel from Expert data

Deep Reinforcement Learning Based Tracking Control of an Autonomous Surface Vessel in Natural Waters

Multiphase Autonomous Docking via Model-Based and Hierarchical Reinforcement Learning

Enhancing UAV Aerial Docking: A Hybrid Approach Combining Offline and Online Reinforcement Learning

Path-Following Control of Unmanned Underwater Vehicle Based on an Improved TD3 Deep Reinforcement Learning

Sim-to-Real Transfer of Adaptive Control Parameters for AUV Stabilization under Current Disturbance

Deep Reinforcement Learning for Vectored Thruster Autonomous Underwater Vehicle Control

Deep reinforcement learning for adaptive path planning and control of an autonomous underwater vehicle

Object Manipulation in Marine Environments using Reinforcement Learning