Abstract:This work proposes an innovative path-following control method, anchored in deep reinforcement learning (DRL), for unmanned underwater vehicles (UUVs). This approach is driven by several new designs, all of which aim to enhance learning efficiency and effectiveness and achieve high-performance UUV control. Specifically, a novel experience replay strategy is designed and integrated within the twin-delayed deep deterministic policy gradient algorithm (TD3). It distinguishes the significance of stored transitions by making a trade-off between rewards and temporal-difference (TD) errors, thus enabling the UUV agent to explore optimal control policies more efficiently. Another major challenge within this control problem arises from action oscillations associated with DRL policies. This issue leads to excessive system wear on actuators and makes real-time application difficult. To mitigate this challenge, a newly improved regularization method is proposed, which provides a moderate level of smoothness to the control policy. Furthermore, a dynamic reward function featuring adaptive constraints is designed to avoid unproductive exploration and expedite learning convergence speed further. Simulation results show that our method garners higher rewards in fewer training episodes compared with mainstream DRL-based control approaches (e.g., deep deterministic policy gradient (DDPG) and vanilla TD3) in UUV applications. Moreover, it can adapt to varying path configurations amid uncertainties and disturbances, all while ensuring high tracking accuracy. Simulation and experimental studies are conducted to verify the effectiveness.

Path Following Optimization for an Underactuated USV Using Smoothly-Convergent Deep Reinforcement Learning

Multi-path Following for Underactuated USV Based on Deep Reinforcement Learning

Path Following Control with Sideslip Reduction for Underactuated Unmanned Surface Vehicles

Path Following Control for Unmanned Surface Vehicles: A Reinforcement Learning-Based Method with Experimental Validation.

LSTM-DPPO based deep reinforcement learning controller for path following optimization of unmanned surface vehicle

An Offline Reinforcement Learning Approach for Path Following of an Unmanned Surface Vehicle

Path-Following Control of Unmanned Underwater Vehicle Based on an Improved TD3 Deep Reinforcement Learning

Deep Interactive Reinforcement Learning for Path Following of Autonomous Underwater Vehicle

USV Formation and Path-Following Control via Deep Reinforcement Learning With Random Braking

Neural Network Model-Based Reinforcement Learning Control for AUV 3-D Path Following

AUV Path Following Control using Deep Reinforcement Learning Under the Influence of Ocean Currents.

Deep Reinforcement Learning-Based Path Planning of Underactuated Surface Vessels

Obstacle avoidance USV in multi-static obstacle environments based on a deep reinforcement learning approach

Data-driven Path Following of Unmanned Surface Vehicles Based on Model-Based Reinforcement Learning and Model Predictive Path Integral Control

Path Planning for Underactuated Unmanned Surface Vehicle Swarm Based on Deep Reinforcement Learning

Path Following Method for AUV Based on Q-Learning and RBF Neural Network

A path planning strategy unified with a COLREGS collision avoidance function based on deep reinforcement learning and artificial potential field

Continuous Control for Autonomous Underwater Vehicle Path Following Using Deep Interactive Reinforcement Learning

Deep reinforcement learning-based controller for dynamic positioning of an unmanned surface vehicle

Model-based Deep Reinforcement Learning for Data-Driven Motion Control of an Under-Actuated Unmanned Surface Vehicle: Path Following and Trajectory Tracking

A Real-time Algorithm for USV Navigation Based on Deep Reinforcement Learning