Abstract:Reinforcement learning (RL) has shown superior performance in solving sequential decision problems. In recent years, RL is gradually being used to solve unmanned driving collision avoidance decision-making problems in complex scenarios. However, ships encounter many scenarios, and the differences in scenarios will seriously hinder the application of RL in collision avoidance at sea. Moreover, the iterative speed of trial-and-error learning for RL in multi-ship encounter scenarios is slow. To solve this problem, this study develops a novel intelligent collision avoidance algorithm based on approximate representation reinforcement learning (AR-RL) to realize the collision avoidance of maritime autonomous surface ships (MASS) in a continuous state space environment involving interactive learning capability like a crew in navigation situation. The new algorithm uses an approximate representation model to deal with the optimization of collision avoidance strategies in a dynamic target encounter situation. The model is combined with prior knowledge and International Regulations for Preventing Collisions at Sea (COLREGs) for optimal performance. This is followed by a design of an online solution to a value function approximation model based on gradient descent. This approach can solve the problem of large-scale collision avoidance policy learning in static-dynamic obstacles mixed environment. Finally, algorithm tests were constructed though two scenarios (i.e., the coastal static obstacle environment and the static-dynamic obstacles mixed environment) using Tianjin Port as an example and compared with multiple groups of algorithms. The results show that the algorithm can improve the large-scale learning efficiency of continuous state space of dynamic obstacle environment by approximate representation. At the same time, the MASS can efficiently and safely avoid obstacles enroute to reaching its target destination. It therefore makes significant contributions to ensuring safety at sea in a mixed traffic involving both manned and MASS in near future.

Ship Collision Avoidance Using Constrained Deep Reinforcement Learning

Ship collision risk analysis: Modeling, visualization and prediction

Deep Reinforcement Learning Based Path Planning and Collision Avoidance for Smart Ships in Complex Environments

Deep Reinforcement Learning Based Collision Avoidance System for Autonomous Ships

Intelligent Ship Collision Avoidance Algorithm Based on DDQN with Prioritized Experience Replay under COLREGs

A Multi-Stage Collision Avoidance Model for Autonomous Ship Based on Fuzzy Set Theory with TL-DDQN Algorithm

A COLREGs-Compliant Deep Reinforcement Learning Approach

A human-like collision avoidance method for autonomous ship with attention-based deep reinforcement learning

Research on Unmanned Surface Vehicle Collision Avoidance Based on Deep Reinforcement Learning

Adaptive Collision Avoidance Decisions in Autonomous Ship Encounter Scenarios Through Rule-Guided Vision Supervised Learning

Collision avoidance for autonomous ship using deep reinforcement learning and prior-knowledge-based approximate representation

Optimizing Multi-Vessel Collision Avoidance Decision Making for Autonomous Surface Vessels: A COLREGs-Compliant Deep Reinforcement Learning Approach

Automatic ship collision avoidance using deep reinforcement learning with LSTM in continuous action spaces

A path planning strategy unified with a COLREGS collision avoidance function based on deep reinforcement learning and artificial potential field

Improved reinforcement learning for collision-free local path planning of dynamic obstacle

An Autonomous Decision-making Algorithm for Ship Collision Avoidance Based on DDQN with Prioritized Experience Replay

Knowledge transfer enabled reinforcement learning for efficient and safe autonomous ship collision avoidance

Improved DQN for Dynamic Obstacle Avoidance and Ship Path Planning

Path Planning Technology of Unmanned Vehicle Based on Improved Deep Reinforcement Learning

COLERGs-constrained Safe Reinforcement Learning for Realising MASS's Risk-Informed Collision Avoidance Decision Making

A Novel Deep Reinforcement Learning for POMDP-based Autonomous Ship Collision Decision-Making