Abstract:The autonomous navigation and obstacle avoidance capabilities of autonomous underwater vehicles (AUVs) are essential for ensuring their safe navigation and long-term, efficient operation. However, the complexity of the marine environment poses significant challenges to safe and effective obstacle avoidance. To address this issue, this study proposes an AUV obstacle avoidance control algorithm based on offline reinforcement learning. This method adopts the Conservative Q-learning (CQL) algorithm, which is based on the Soft Actor-Critic (SAC) framework. It learns from obtained historical obstacle avoidance data and ultimately achieves a favorable obstacle avoidance control strategy. In this method, PID and SAC control algorithms are utilized to generate expert obstacle avoidance data to construct a diversified offline database. Additionally, based on the line-of-sight (LOS) guidance method and artificial potential field (APF) method, information regarding the distance and orientation of targets and obstacles is incorporated into the state space, and heading and obstacle avoidance reward terms are integrated into the reward function design. The algorithm successfully guides the AUV in autonomous navigation and dynamic obstacle avoidance in three-dimensional space. Furthermore, the algorithm exhibits a certain degree of anti-interference capability against uncertain disturbances and ocean currents, enhancing the safety and robustness of the AUV system. Simulation results fully demonstrate the feasibility and effectiveness of the intelligent obstacle avoidance method based on offline reinforcement learning. This study highlights the profound significance of offline reinforcement learning in enabling robust and reliable control systems for AUVs, paving the way for enhanced operational capabilities in challenging marine environments.

Reinforcement Learning Based Parameter Optimization of Active Disturbance Rejection Control for Autonomous Underwater Vehicle

An Underactuated AUV Tracking Algorithm Based on Backstepping Adaptive Sliding Mode Control

ADRC-Based Underwater Navigation Control and Parameter Tuning of an Amphibious Multirotor Vehicle

Single degree of freedom control based on deep reinforcement learning for underwater unmanned vehicle

Anti-disturbance Depth Control Based on the Robust $$L_{1}$$ Filter for Under-Actuated AUV with Switched Linear Parameter Varying Dynamics

Neural Network Model-Based Reinforcement Learning Control for AUV 3-D Path Following

Motion Control of Autonomous Underwater Helicopter Based on Linear Active Disturbance Rejection Control with Tracking Differentiator

Reinforcement Learning Based Obstacle Avoidance for Autonomous Underwater Vehicle

Soft Actor–Critic Based Active Disturbance Rejection Path Following Control for Unmanned Surface Vessel under Wind and Wave Disturbances

An adaptive PID controller for path following of autonomous underwater vehicle based on Soft Actor–Critic

Sim-to-Real Transfer of Adaptive Control Parameters for AUV Stabilization under Current Disturbance

Depth Control of an Oil Bladder Type Deep-Sea AUV Based on Fuzzy Adaptive Linear Active Disturbance Rejection Control

An Adaptive Sliding Mode Control Strategy for the Heading Control of Autonomous Underwater Vehicles

A Novel Adaptive Backstepping Sliding Mode Control for a Lightweight Autonomous Underwater Vehicle with Input Saturation

Learning-based Tracking Control of AUV: Mixed Policy Improvement and Game-Based Disturbance Rejection

ADRC‐SMC‐based disturbance rejection depth‐tracking control of underactuated AUV

Sim-to-real transfer of adaptive control parameters for AUV stabilisation under current disturbance

Research on obstacle avoidance of underactuated autonomous underwater vehicle based on offline reinforcement learning

Double-loop LQR depth tracking control of underactuated AUV: Methodology and comparative experiments

Adaptive Optimal Tracking Control for Uncertain Unmanned Surface Vessel Via Reinforcement Learning

Attitude Control of Underwater Glider Combined Reinforcement Learning with Active Disturbance Rejection Control