Abstract:This work proposes an innovative path-following control method, anchored in deep reinforcement learning (DRL), for unmanned underwater vehicles (UUVs). This approach is driven by several new designs, all of which aim to enhance learning efficiency and effectiveness and achieve high-performance UUV control. Specifically, a novel experience replay strategy is designed and integrated within the twin-delayed deep deterministic policy gradient algorithm (TD3). It distinguishes the significance of stored transitions by making a trade-off between rewards and temporal-difference (TD) errors, thus enabling the UUV agent to explore optimal control policies more efficiently. Another major challenge within this control problem arises from action oscillations associated with DRL policies. This issue leads to excessive system wear on actuators and makes real-time application difficult. To mitigate this challenge, a newly improved regularization method is proposed, which provides a moderate level of smoothness to the control policy. Furthermore, a dynamic reward function featuring adaptive constraints is designed to avoid unproductive exploration and expedite learning convergence speed further. Simulation results show that our method garners higher rewards in fewer training episodes compared with mainstream DRL-based control approaches (e.g., deep deterministic policy gradient (DDPG) and vanilla TD3) in UUV applications. Moreover, it can adapt to varying path configurations amid uncertainties and disturbances, all while ensuring high tracking accuracy. Simulation and experimental studies are conducted to verify the effectiveness.

Deep Reinforcement Learning-Based Path Control and Optimization for Unmanned Ships

An Autonomous Path Planning Model for Unmanned Ships Based on Deep Reinforcement Learning

Sail-rudder Collaborative Control of Unmanned Sailboat Based on Reinforcement Learning

Deep Reinforcement Learning Based Path Planning and Collision Avoidance for Smart Ships in Complex Environments

A knowledge-free path planning approach for smart ships based on reinforcement learning

Path Planning of Maritime Autonomous Surface Ships in Unknown Environment with Reinforcement Learning

Neural Network Model-Based Reinforcement Learning Control for AUV 3-D Path Following

Path Planning of Unmanned Underwater Vehicles Based on Deep Reinforcement Learning Algorithm

Path Planning Algorithm for Unmanned Surface Vessel Based on Multiobjective Reinforcement Learning

A path planning strategy unified with a COLREGS collision avoidance function based on deep reinforcement learning and artificial potential field

Path Planning based on Deep Reinforcement Learning for Autonomous Underwater Vehicles under Ocean Current Disturbance

Deep reinforcement learning with dynamic window approach based collision avoidance path planning for maritime autonomous surface ships

LSTM-DPPO based deep reinforcement learning controller for path following optimization of unmanned surface vehicle

Using Deep Reinforcement Learning Methods for Autonomous Vessels in 2D Environments

Path-Following Control of Unmanned Underwater Vehicle Based on an Improved TD3 Deep Reinforcement Learning

Path planning of autonomous underwater vehicle in unknown environment based on improved deep reinforcement learning

An autonomous coverage path planning algorithm for maritime search and rescue of persons-in-water based on deep reinforcement learning

AUV Path Planning with Kinematic Constraints in Unknown Environment Using Reinforcement Learning.

Deep Interactive Reinforcement Learning for Path Following of Autonomous Underwater Vehicle

Harnessing traditional controllers for fast-track training of deep reinforcement learning control strategies

Research on Path Planning based on Unmanned Ship