Deep Reinforcement Learning with Model Predictive Control for Path Following of Autonomous Underwater Vehicle

Zheng Zhang,Xingwei Pan,Tianhao Chen,Dong Jiang,Zheng Fang,Guangliang Li

DOI: https://doi.org/10.23919/ccc63176.2024.10661914

2024-01-01

Abstract:Autonomous underwater vehicle (AUV) is becoming increasingly important to perform underwater tasks. Model-free reinforcement learning was applied for AUV path following, but is usually inefficient in sampling and time-consuming for training control policies. This paper proposed and implemented an MPC-based model-based reinforcement learning (MB-MPC) method for AUV path following. To better capture the uncertainty of the dynamics model, feature vectors from the penultimate layer of the neural network fitting the dynamics model were extracted and Bayesian linear regression was performed on the extracted feature vectors and sampled. Model predictive control and cross-entropy method were used to select an optimal sequence of actions for each AUV state to perform. The proposed MB-MPC method was tested with an AUV simulator on Gazebo in path following tasks with and without currents. Our simulation results show that MB-MPC allows control policy of an AUV to obtain a better performance faster than traditional deep model-free reinforcement learning and model-based reinforcement learning without MPC. Further analysis indicates that MB-MPC has better generalization and adaptation to new tasks than model-free methods and traditional PID controller.