Abstract:This work proposes an innovative path-following control method, anchored in deep reinforcement learning (DRL), for unmanned underwater vehicles (UUVs). This approach is driven by several new designs, all of which aim to enhance learning efficiency and effectiveness and achieve high-performance UUV control. Specifically, a novel experience replay strategy is designed and integrated within the twin-delayed deep deterministic policy gradient algorithm (TD3). It distinguishes the significance of stored transitions by making a trade-off between rewards and temporal-difference (TD) errors, thus enabling the UUV agent to explore optimal control policies more efficiently. Another major challenge within this control problem arises from action oscillations associated with DRL policies. This issue leads to excessive system wear on actuators and makes real-time application difficult. To mitigate this challenge, a newly improved regularization method is proposed, which provides a moderate level of smoothness to the control policy. Furthermore, a dynamic reward function featuring adaptive constraints is designed to avoid unproductive exploration and expedite learning convergence speed further. Simulation results show that our method garners higher rewards in fewer training episodes compared with mainstream DRL-based control approaches (e.g., deep deterministic policy gradient (DDPG) and vanilla TD3) in UUV applications. Moreover, it can adapt to varying path configurations amid uncertainties and disturbances, all while ensuring high tracking accuracy. Simulation and experimental studies are conducted to verify the effectiveness.

LSTM-DPPO based deep reinforcement learning controller for path following optimization of unmanned surface vehicle

Deep reinforcement learning-based controller for dynamic positioning of an unmanned surface vehicle

Path-Following Control of Unmanned Underwater Vehicle Based on an Improved TD3 Deep Reinforcement Learning

Path Following Control for Unmanned Surface Vehicles: A Reinforcement Learning-Based Method with Experimental Validation.

Robust Motion Control for UAV in Dynamic Uncertain Environments Using Deep Reinforcement Learning.

Path Following for Autonomous Ground Vehicle Using DDPG Algorithm: A Reinforcement Learning Approach

Path Following Optimization for an Underactuated USV Using Smoothly-Convergent Deep Reinforcement Learning

Neural Network Model-Based Reinforcement Learning Control for AUV 3-D Path Following

Deep Reinforcement Learning Approach with Multiple Experience Pools for UAV's Autonomous Motion Planning in Complex Unknown Environments

A Learning-Based Flexible Autonomous Motion Control Method for UAV in Dynamic Unknown Environments

Multi-path Following for Underactuated USV Based on Deep Reinforcement Learning

Path-Tracking Control Strategy of Unmanned Vehicle Based on DDPG Algorithm

Data-Driven Performance-Prescribed Reinforcement Learning Control of an Unmanned Surface Vehicle

Path Planning Technology of Unmanned Vehicle Based on Improved Deep Reinforcement Learning

Continuous Control for Autonomous Underwater Vehicle Path Following Using Deep Interactive Reinforcement Learning

An Offline Reinforcement Learning Approach for Path Following of an Unmanned Surface Vehicle

Model-based Deep Reinforcement Learning for Data-Driven Motion Control of an Under-Actuated Unmanned Surface Vehicle: Path Following and Trajectory Tracking

AUV Path Following Control using Deep Reinforcement Learning Under the Influence of Ocean Currents.

Path Planning of Unmanned Underwater Vehicles Based on Deep Reinforcement Learning Algorithm

Speed and Heading Control of an Unmanned Surface Vehicle Using Deep Reinforcement Learning

AUV position tracking and trajectory control based on fast-deployed deep reinforcement learning method