Abstract:Reinforcement learning (RL) has shown superior performance in solving sequential decision problems. In recent years, RL is gradually being used to solve unmanned driving collision avoidance decision-making problems in complex scenarios. However, ships encounter many scenarios, and the differences in scenarios will seriously hinder the application of RL in collision avoidance at sea. Moreover, the iterative speed of trial-and-error learning for RL in multi-ship encounter scenarios is slow. To solve this problem, this study develops a novel intelligent collision avoidance algorithm based on approximate representation reinforcement learning (AR-RL) to realize the collision avoidance of maritime autonomous surface ships (MASS) in a continuous state space environment involving interactive learning capability like a crew in navigation situation. The new algorithm uses an approximate representation model to deal with the optimization of collision avoidance strategies in a dynamic target encounter situation. The model is combined with prior knowledge and International Regulations for Preventing Collisions at Sea (COLREGs) for optimal performance. This is followed by a design of an online solution to a value function approximation model based on gradient descent. This approach can solve the problem of large-scale collision avoidance policy learning in static-dynamic obstacles mixed environment. Finally, algorithm tests were constructed though two scenarios (i.e., the coastal static obstacle environment and the static-dynamic obstacles mixed environment) using Tianjin Port as an example and compared with multiple groups of algorithms. The results show that the algorithm can improve the large-scale learning efficiency of continuous state space of dynamic obstacle environment by approximate representation. At the same time, the MASS can efficiently and safely avoid obstacles enroute to reaching its target destination. It therefore makes significant contributions to ensuring safety at sea in a mixed traffic involving both manned and MASS in near future.

Reinforcement Learning Based Bi-stage Control for Semi-enclosed Berth

Collision Avoidance and Path Point Tracking Control for Underactuated Unmanned Surface Vehicles with Unknown Model Nonlinearity

Safe deep reinforcement learning-based adaptive control for USV interception mission

Model Reference Adaptive Control-Based Autonomous Berthing of an Unmanned Surface Vehicle under Environmental Disturbance

Automatic Berthing System for a Ship and Study on a Hybrid Control for the System

Automatic berthing of unmanned surface vessels with predetermined performance

Using Deep Reinforcement Learning Methods for Autonomous Vessels in 2D Environments

A path planning strategy unified with a COLREGS collision avoidance function based on deep reinforcement learning and artificial potential field

Real-time Planning and Collision Avoidance Control Method Based on Deep Reinforcement Learning

Deep Reinforcement Learning Based Path Planning and Collision Avoidance for Smart Ships in Complex Environments

Collision-free automatic berthing of maritime autonomous surface ships via safety-certified active disturbance rejection control

Adaptive Neural Network Iterative Sliding Mode Course Tracking Control for Unmanned Surface Vessels

A Novel Reinforcement Learning Collision Avoidance Algorithm for USVs Based on Maneuvering Characteristics and COLREGs

2-Level Reinforcement Learning for Ships on Inland Waterways: Path Planning and Following

A COLREGs-Compliant Deep Reinforcement Learning Approach

Fuzzy Categorical Deep Reinforcement Learning of a Defensive Game for an Unmanned Surface Vessel

Graph Search-Based Path Planning for Automatic Ship Berthing

AUV Path Planning with Kinematic Constraints in Unknown Environment Using Reinforcement Learning.

Collision avoidance for autonomous ship using deep reinforcement learning and prior-knowledge-based approximate representation

Barrier Lyapunov function-based adaptive optimized control for full-state and input-constrained dynamic positioning of marine vessels with simulation and model-scale tests

Collision probability reduction method for tracking control in automatic docking / berthing using reinforcement learning