Abstract:This work proposes an innovative path-following control method, anchored in deep reinforcement learning (DRL), for unmanned underwater vehicles (UUVs). This approach is driven by several new designs, all of which aim to enhance learning efficiency and effectiveness and achieve high-performance UUV control. Specifically, a novel experience replay strategy is designed and integrated within the twin-delayed deep deterministic policy gradient algorithm (TD3). It distinguishes the significance of stored transitions by making a trade-off between rewards and temporal-difference (TD) errors, thus enabling the UUV agent to explore optimal control policies more efficiently. Another major challenge within this control problem arises from action oscillations associated with DRL policies. This issue leads to excessive system wear on actuators and makes real-time application difficult. To mitigate this challenge, a newly improved regularization method is proposed, which provides a moderate level of smoothness to the control policy. Furthermore, a dynamic reward function featuring adaptive constraints is designed to avoid unproductive exploration and expedite learning convergence speed further. Simulation results show that our method garners higher rewards in fewer training episodes compared with mainstream DRL-based control approaches (e.g., deep deterministic policy gradient (DDPG) and vanilla TD3) in UUV applications. Moreover, it can adapt to varying path configurations amid uncertainties and disturbances, all while ensuring high tracking accuracy. Simulation and experimental studies are conducted to verify the effectiveness.

Target following for an autonomous underwater vehicle using regularized ELM-based reinforcement learning

Deep Interactive Reinforcement Learning for Path Following of Autonomous Underwater Vehicle

Neural Network Model-Based Reinforcement Learning Control for AUV 3-D Path Following

Continuous Control for Autonomous Underwater Vehicle Path Following Using Deep Interactive Reinforcement Learning

Target Search Control Of Auv In Underwater Environment With Deep Reinforcement Learning

Path-Following Control of Unmanned Underwater Vehicle Based on an Improved TD3 Deep Reinforcement Learning

Path planning of autonomous underwater vehicle in unknown environment based on improved deep reinforcement learning

AUV Obstacle Avoidance Framework Based on Event-Triggered Reinforcement Learning

Attention-Based Meta-Reinforcement Learning for Tracking Control of AUV with Time-Varying Dynamics

A Modified ALOS Method of Path Tracking for AUVs with Reinforcement Learning Accelerated by Dynamic Data-Driven AUV Model

AUV Path Following Control using Deep Reinforcement Learning Under the Influence of Ocean Currents.

A General Motion Controller Based on Deep Reinforcement Learning for an Autonomous Underwater Vehicle with Unknown Disturbances.

Guidance and Control of Autonomous Surface Underwater Vehicles for Target Tracking in Ocean Environment by Deep Reinforcement Learning

Three-Dimensional Path-Following Control of an Autonomous Underwater Vehicle Based on Deep Reinforcement Learning

Underwater Multi-agent Cooperative Formation Hunting Based on Deep Reinforcement Learning

Path Following Control for Unmanned Surface Vehicles: A Reinforcement Learning-Based Method with Experimental Validation.

A reinforcement learning path planning approach for range-only underwater target localization with autonomous vehicles

Deep Reinforcement Learning for Vectored Thruster Autonomous Underwater Vehicle Control

Intelligent AUV Surfacing Control in Network Attack Scenario

Generative adversarial interactive imitation learning for path following of autonomous underwater vehicle

Heading Control for an Autonomous Underwater Vehicle Using ELM-based Q-learning