Abstract:This paper investigates trajectory tracking problem for a class of underactuated autonomous underwater vehicles (AUVs) with unknown dynamics and constrained inputs. Different from existing policy gradient methods which employ single actor-critic but cannot realize satisfactory tracking control accuracy and stable learning, our proposed algorithm can achieve high-level tracking control accuracy of AUVs and stable learning by applying a hybrid actors-critics architecture, where multiple actors and critics are trained to learn a deterministic policy and action-value function, respectively. Specifically, for the critics, the expected absolute Bellman error based updating rule is used to choose the worst critic to be updated in each time step. Subsequently, to calculate the loss function with more accurate target value for the chosen critic, Pseudo Q-learning, which uses sub-greedy policy to replace the greedy policy in Q-learning, is developed for continuous action spaces, and Multi Pseudo Q-learning (MPQ) is proposed to reduce the overestimation of action-value function and to stabilize the learning. As for the actors, deterministic policy gradient is applied to update the weights, and the final learned policy is defined as the average of all actors to avoid large but bad updates. Moreover, the stability analysis of the learning is given qualitatively. The effectiveness and generality of the proposed MPQ-based Deterministic Policy Gradient (MPQ-DPG) algorithm are verified by the application on AUV with two different reference trajectories. And the results demonstrate high-level tracking control accuracy and stable learning of MPQ-DPG. Besides, the results also validate that increasing the number of the actors and critics will further improve the performance.

Trajectory Tracking of an Underwater Glider in Current Based on Deep Reinforcement Learning

Path planning for underwater gliders in time-varying ocean current using deep reinforcement learning

Trajectory Tracking Control for Subsea Cruise and Docking of Autonomous Underwater Helicopter

Gliding in extreme waters: Dynamic Modeling and Nonlinear Control of an Agile Underwater Glider

Deep reinforcement learning-based path planning method for underwater gliders in unknown 3D marine environment

Neural Network Model-Based Reinforcement Learning Control for AUV 3-D Path Following

Target Tracking Control of a Biomimetic Underwater Vehicle Through Deep Reinforcement Learning

Multi Pseudo Q-learning Based Deterministic Policy Gradient for Tracking Control of Autonomous Underwater Vehicles

Surface path tracking method of autonomous surface underwater vehicle based on deep reinforcement learning

Trajectory tracking control of vectored thruster autonomous underwater vehicles based on deep reinforcement learning

AUV position tracking and trajectory control based on fast-deployed deep reinforcement learning method

High-Level Tracking of Autonomous Underwater Vehicles Based on Pseudo Averaged Q-Learning.

PID Controller Based on Improved DDPG for Trajectory Tracking Control of USV

Single degree of freedom control based on deep reinforcement learning for underwater unmanned vehicle

Backstepping control of gliding robotic fish for pitch and 3D trajectory tracking

USV Trajectory Tracking Control Based on Receding Horizon Reinforcement Learning

Prioritized experience replay based reinforcement learning for adaptive tracking control of autonomous underwater vehicle

Attention-Based Meta-Reinforcement Learning for Tracking Control of AUV with Time-Varying Dynamics

Plume Tracing Via Model-Free Reinforcement Learning Method

AUV Path Following Control using Deep Reinforcement Learning Under the Influence of Ocean Currents.

Adaptive integral terminal sliding mode based trajectory tracking control of underwater glider