Abstract:The International Journal of Robotics Research, Ahead of Print. Learning-based adaptive control methods hold the potential to empower autonomous agents in mitigating the impact of process variations with minimal human intervention. However, their application to autonomous underwater vehicles (AUVs) has been constrained by two main challenges: (1) the presence of unknown dynamics in the form of sea current disturbances, which cannot be modelled or measured due to limited sensor capability, particularly on smaller low-cost AUVs, and (2) the nonlinearity of AUV tasks, where the controller response at certain operating points must be excessively conservative to meet specifications at other points. Deep Reinforcement Learning (DRL) offers a solution to these challenges by training versatile neural network policies. Nevertheless, the application of DRL algorithms to AUVs has been predominantly limited to simulated environments due to their inherent high sample complexity and the distribution shift problem. This paper introduces a novel approach by combining the Maximum Entropy Deep Reinforcement Learning framework with a classic model-based control architecture to formulate an adaptive controller. In this framework, we propose a Sim-to-Real transfer strategy, incorporating a bio-inspired experience replay mechanism, an enhanced domain randomisation technique, and an evaluation protocol executed on a physical platform. Our experimental assessments demonstrate the effectiveness of this method in learning proficient policies from suboptimal simulated models of the AUV. When transferred to a real-world vehicle, the approach exhibits a control performance three times higher compared to its model-based nonadaptive but optimal counterpart.

A selected review on reinforcement learning based control for autonomous underwater vehicles

Neural Network Model-Based Reinforcement Learning Control for AUV 3-D Path Following

Deep Reinforcement Learning for Vectored Thruster Autonomous Underwater Vehicle Control

Deep Interactive Reinforcement Learning for Path Following of Autonomous Underwater Vehicle

Application of Deep Reinforcement Learning in UAVs: A Review

Deep Reinforcement Learning Based Optimal Trajectory Tracking Control of Autonomous Underwater Vehicle

AUV Obstacle Avoidance Framework Based on Event-Triggered Reinforcement Learning

A General Motion Controller Based on Deep Reinforcement Learning for an Autonomous Underwater Vehicle with Unknown Disturbances.

End-To-End Sensorimotor Control Problems Of Auvs With Deep Reinforcement Learning

AUV Path Following Control using Deep Reinforcement Learning Under the Influence of Ocean Currents.

Target Search Control Of Auv In Underwater Environment With Deep Reinforcement Learning

Attention-Based Meta-Reinforcement Learning for Tracking Control of AUV with Time-Varying Dynamics

Guidance and Control of Autonomous Surface Underwater Vehicles for Target Tracking in Ocean Environment by Deep Reinforcement Learning

Reinforcement Learning for Autonomous Underwater Vehicles Via Data-Informed Domain Randomization

Deep Reinforcement Learning Based Tracking Control of an Autonomous Surface Vessel in Natural Waters

Intelligent AUV Surfacing Control in Network Attack Scenario

General reinforcement learning control for AUV manoeuvring in turbulent flows

Intelligent Trajectory Tracking and Formation Control of Underactuated Autonomous Underwater Vehicles: A Critical Review

Depth Control of Model-Free AUVs Via Reinforcement Learning

Sim-to-real transfer of adaptive control parameters for AUV stabilisation under current disturbance