Abstract:Reinforcement learning (RL) has shown superior performance in solving sequential decision problems. In recent years, RL is gradually being used to solve unmanned driving collision avoidance decision-making problems in complex scenarios. However, ships encounter many scenarios, and the differences in scenarios will seriously hinder the application of RL in collision avoidance at sea. Moreover, the iterative speed of trial-and-error learning for RL in multi-ship encounter scenarios is slow. To solve this problem, this study develops a novel intelligent collision avoidance algorithm based on approximate representation reinforcement learning (AR-RL) to realize the collision avoidance of maritime autonomous surface ships (MASS) in a continuous state space environment involving interactive learning capability like a crew in navigation situation. The new algorithm uses an approximate representation model to deal with the optimization of collision avoidance strategies in a dynamic target encounter situation. The model is combined with prior knowledge and International Regulations for Preventing Collisions at Sea (COLREGs) for optimal performance. This is followed by a design of an online solution to a value function approximation model based on gradient descent. This approach can solve the problem of large-scale collision avoidance policy learning in static-dynamic obstacles mixed environment. Finally, algorithm tests were constructed though two scenarios (i.e., the coastal static obstacle environment and the static-dynamic obstacles mixed environment) using Tianjin Port as an example and compared with multiple groups of algorithms. The results show that the algorithm can improve the large-scale learning efficiency of continuous state space of dynamic obstacle environment by approximate representation. At the same time, the MASS can efficiently and safely avoid obstacles enroute to reaching its target destination. It therefore makes significant contributions to ensuring safety at sea in a mixed traffic involving both manned and MASS in near future.

Harnessing traditional controllers for fast-track training of deep reinforcement learning control strategies

Navigating the Ocean with DRL: Path following for marine vessels

Deep Reinforcement Learning Based Tracking Control of an Autonomous Surface Vessel in Natural Waters

AI on the Water: Applying DRL to Autonomous Vessel Navigation

Learning with Training Wheels: Speeding up Training with a Simple Controller for Deep Reinforcement Learning

Comparison of path following in ships using modern and traditional controllers

A Novel Path Following Approach for Autonomous Ships Based on Fast Marching Method and Deep Reinforcement Learning

Improved Reinforcement Learning For Cargo Ship Steering Control

2-Level Reinforcement Learning for Ships on Inland Waterways: Path Planning and Following

An Autonomous Path Planning Model for Unmanned Ships Based on Deep Reinforcement Learning

Sim-to-real transfer of adaptive control parameters for AUV stabilisation under current disturbance

Deep Reinforcement Learning-Based Path Control and Optimization for Unmanned Ships

Deep Reinforcement Learning Controller for 3D Path Following and Collision Avoidance by Autonomous Underwater Vehicles

Deep reinforcement learning-based controller for dynamic positioning of an unmanned surface vehicle

Sim-to-Real Transfer of Adaptive Control Parameters for AUV Stabilization under Current Disturbance

Deep Reinforcement Learning Based Collision Avoidance System for Autonomous Ships

Data-Driven Performance-Prescribed Reinforcement Learning Control of an Unmanned Surface Vehicle

Deep Reinforcement Learning for Vectored Thruster Autonomous Underwater Vehicle Control

Controlling a cargo ship without human experience using deep Q-network

Optimizing Multi-Vessel Collision Avoidance Decision Making for Autonomous Surface Vessels: A COLREGs-Compliant Deep Reinforcement Learning Approach

Collision avoidance for autonomous ship using deep reinforcement learning and prior-knowledge-based approximate representation