Abstract:To address the challenge of partially observable environment states in multi-ship collision avoidance decision-making, a novel collision avoidance decision model is developed based on Partially Observable Markov Decision Processes (POMDP), integrating both static and dynamic obstacles. A new reward mechanism is designed to overcome the problem of sparse rewards, incorporating the international regulations for preventing collisions at sea (COLREGs) into the reward function. The Proximal Policy Optimization (PPO) algorithm is employed to train the model, accompanied by a suitable network structure. Furthermore, image conversion and scaling operations are applied to the network’s architecture to reduce the dimensionality of the state space, thereby enhancing the training speed and fitting performance of the algorithm. Subsequently, a simulation environment is created using the Gym platform, incorporating multiple static and dynamic obstacles, and a series of experiments containing classic encounter scenarios are designed to validate the model. The evaluation metrics, such as the total cumulative reward and training steps, demonstrate that the proposed algorithm is capable of making accurate decisions in an environment with combined static and dynamic obstacles. Through several ablation experiments and analysis of total run-time, it is shown that the proposed reward mechanism effectively addresses the issue of sparse rewards, while the algorithm significantly improves the fitting speed and expedites the training process. The proposed algorithm’s ability to address the problem of partially observable environment states in multi-ship collision avoidance decision-making is confirmed through a comparison with other classical deep reinforcement learning (DRL) algorithms.

Intelligent Ship Decision System Based on DDPG Algorithm

An Autonomous Path Planning Model for Unmanned Ships Based on Deep Reinforcement Learning

A knowledge-free path planning approach for smart ships based on reinforcement learning

Intelligent Smart Marine Autonomous Surface Ship Decision System Based on Improved PPO Algorithm

A Novel Path Planning Approach For Unmanned Ships Based On Deep Reinforcement Learning

Intelligent Decision-Making System for Multiple Marine Autonomous Surface Ships Based on Deep Reinforcement Learning

Adaptive Path Planning for Autonomous Ships Based on Deep Reinforcement Learning Combined with Images

A Novel Path Following Approach for Autonomous Ships Based on Fast Marching Method and Deep Reinforcement Learning

Deep Reinforcement Learning Based Path Planning and Collision Avoidance for Smart Ships in Complex Environments

Path Planning of Maritime Autonomous Surface Ships in Unknown Environment with Reinforcement Learning

A Novel Deep Reinforcement Learning for POMDP-based Autonomous Ship Collision Decision-Making

Deep Reinforcement Learning-Based Path Control and Optimization for Unmanned Ships

An Improved Deep Reinforcement Learning Method-Based Nonparametric Modeling of Ship Dynamics

Research and Design of an Autonomous Underwater Vehicle Path Planning Method Based on Deep Reinforcement Learning

Intelligent ship anti-rolling control system based on a deep deterministic policy gradient algorithm and the Magnus effect

A partially observable multi-ship collision avoidance decision-making model based on deep reinforcement learning

Decision-Making for the Autonomous Navigation of Maritime Autonomous Surface Ships Based on Scene Division and Deep Reinforcement Learning

AI on the Water: Applying DRL to Autonomous Vessel Navigation

Method for Intelligent Obstacle Avoidance Decision-Making of Unmanned Vessel in Unknown Waters

Navigating the Ocean with DRL: Path following for marine vessels

Research on Intelligent Collision Avoidance Decision-Making of Unmanned Ship in Unknown Environments