Abstract:Autonomous collision avoidance technology provides an intelligent method for unmanned surface vehicles’ (USVs) safe and efficient navigation. In this paper, the USV collision avoidance problem under the constraint of the international regulations for preventing collisions at sea (COLREGs) was studied. Here, a reinforcement learning collision avoidance (RLCA) algorithm is proposed that complies with USV maneuverability. Notably, the reinforcement learning agent does not require any prior knowledge about USV collision avoidance from humans to learn collision avoidance motions well. The double-DQN method was used to reduce the overestimation of the action-value function. A dueling network architecture was adopted to clearly distinguish the difference between a great state and an excellent action. Aiming at the problem of agent exploration, a method based on the characteristics of USV collision avoidance, the category-based exploration method, can improve the exploration ability of the USV. Because a large number of turning behaviors in the early steps may affect the training, a method to discard some of the transitions was designed, which can improve the effectiveness of the algorithm. A finite Markov decision process (MDP) that conforms to the USVs’ maneuverability and COLREGs was used for the agent training. The RLCA algorithm was tested in a marine simulation environment in many different USV encounters, which showed a higher average reward. The RLCA algorithm bridged the divide between USV navigation status information and collision avoidance behavior, resulting in successfully planning a safe and economical path to the terminal.

Adaptive Optimal Surrounding Control of Multiple Unmanned Surface Vessels via Actor-Critic Reinforcement Learning

Target Circumnavigation Control with an Unmanned Surface Vehicle Based on Relative Range Measurements

Cooperative Target Enclosing Control for Multiple Unmanned Surface Vehicles with Unknown Dynamics and Safety Assurance

Neural Network Model-Based Reinforcement Learning Control for AUV 3-D Path Following

Distributed Surrounding Control of Multiple Unmanned Surface Vessels With Varying Interconnection Topologies

Distributed Control of Unmanned Marine Vehicles for Target Circumnavigation in Communication-Denied Environments

Collision Avoidance and Path Point Tracking Control for Underactuated Unmanned Surface Vehicles with Unknown Model Nonlinearity

Model Predictive Control Based on State Space and Risk Augmentation for Unmanned Surface Vessel Trajectory Tracking

Hierarchical Control Design for the Cooperative Target Enclosing Motion of Unmanned Surface Vehicle.

Adaptive optimal formation control for unmanned surface vehicles with guaranteed performance using actor‐critic learning architecture

Self‐learning‐based optimal tracking control of an unmanned surface vehicle with pose and velocity constraints

Simultaneous Control and Guidance of an AUV Based on Soft Actor-Critic

Distributed Model Predictive Contouring Control of Unmanned Surface Vessels

Data-Driven Performance-Prescribed Reinforcement Learning Control of an Unmanned Surface Vehicle

Adaptive Optimal Tracking Control of an Underactuated Surface Vessel Using Actor–Critic Reinforcement Learning

Soft Actor–Critic Based Active Disturbance Rejection Path Following Control for Unmanned Surface Vessel under Wind and Wave Disturbances

Adaptive Dynamic Model-Based Path Following Controller Design for an Unmanned Surface Vessel

An Offline Reinforcement Learning Approach for Path Following of an Unmanned Surface Vehicle

Distributed Optimal Formation Control for Unmanned Surface Vessels by a Regularized Game-Based Approach

Adaptive Neural Network Iterative Sliding Mode Course Tracking Control for Unmanned Surface Vessels

A Novel Reinforcement Learning Collision Avoidance Algorithm for USVs Based on Maneuvering Characteristics and COLREGs