Abstract:A mixed adaptive dynamic programming (ADP) scheme based on zero-sum game theory is developed to address optimal control problems of autonomous underwater vehicle (AUV) systems subject to disturbances and safe constraints. By combining prior dynamic knowledge and actual sampled data, the proposed approach effectively mitigates the defect caused by the inaccurate dynamic model and significantly improves the training speed of the ADP algorithm. Initially, the dataset is enriched with sufficient reference data collected based on a nominal model without considering modelling bias. Also, the control object interacts with the real environment and continuously gathers adequate sampled data in the dataset. To comprehensively leverage the advantages of model-based and model-free methods during training, an adaptive tuning factor is introduced based on the dataset that possesses model-referenced information and conforms to the distribution of the real-world environment, which balances the influence of model-based control law and data-driven policy gradient on the direction of policy improvement. As a result, the proposed approach accelerates the learning speed compared to data-driven methods, concurrently also enhancing the tracking performance in comparison to model-based control methods. Moreover, the optimal control problem under disturbances is formulated as a zero-sum game, and the actor-critic-disturbance framework is introduced to approximate the optimal control input, cost function, and disturbance policy, respectively. Furthermore, the convergence property of the proposed algorithm based on the value iteration method is analysed. Finally, an example of AUV path following based on the improved line-of-sight guidance is presented to demonstrate the effectiveness of the proposed method.

Game-based Distributed Optimal Formation Tracking Control of Underactuated AUVs Based on Reinforcement Learning

Distributed Optimal Formation Tracking Control Based on Reinforcement Learning for Underactuated AUVs with Asymmetric Constraints

Cooperative Learning Formation Control of Multiple Autonomous Underwater Vehicles with Prescribed Performance Based on Position Estimation

Leader-Follower Formation Control of Autonomous Underwater Vehicles

Feedforward/Feedback Controller for Formation Control of Multiple Autonomous Underwater Vehicles

Underwater Multi-agent Cooperative Formation Hunting Based on Deep Reinforcement Learning

Bearing-Based Robust Formation Tracking Control of Underactuated AUVs With Optimal Parameter Tuning

Receding Horizon Formation Tracking Control of Constrained Underactuated Autonomous Underwater Vehicles

Distributed Adaptive Fuzzy 3-D Formation Tracking Control of Underactuated Autonomous Underwater Vehicles

A General Motion Controller Based on Deep Reinforcement Learning for an Autonomous Underwater Vehicle with Unknown Disturbances.

Finite-time Velocity-Observed Based Adaptive Output-Feedback Trajectory Tracking Formation Control for Underactuated Unmanned Underwater Vehicles with Prescribed Transient Performance

Leader–follower Formation Control of Underactuated Autonomous Underwater Vehicles

Formation Trajectory Tracking of Discrete-Time Distributed Multi-AUVs with Nonconvex Control Inputs and Weak Communication

Adaptive Formation Control of Autonomous Underwater Vehicles with Model Uncertainties

Learning-based Robust Optimal Tracking Controller Design for Unmanned Underwater Vehicles with Full-State and Input Constraints

Event-triggered output-feedback adaptive tracking control of autonomous underwater vehicles using reinforcement learning

Bounded Neural Adaptive Formation Control of Multiple Underactuated AUVs under Uncertain Dynamics.

Adaptive Optimal Tracking Control for Uncertain Unmanned Surface Vessel Via Reinforcement Learning

Attention-Based Meta-Reinforcement Learning for Tracking Control of AUV with Time-Varying Dynamics

Learning-based Tracking Control of AUV: Mixed Policy Improvement and Game-Based Disturbance Rejection

Distributed Neurodynamics-Based Backstepping Optimal Control for Robust Constrained Consensus of Underactuated Underwater Vehicles Fleet