Abstract:Unmanned Surface Vehicles technology (USVs) is an exciting topic that essentially deploys an algorithm to safely and efficiently performs a mission. Although reinforcement learning is a well-known approach to modeling such a task, instability and divergence may occur when combining off-policy and function approximation. In this work, we used deep reinforcement learning combining Q-learning with a neural representation to avoid instability. Our methodology uses deep q-learning and combines it with a rolling wave planning approach on agile methodology. Our method contains two critical parts in order to perform missions in an unknown environment. The first is a path planner that is responsible for generating a potential effective path to a destination without considering the details of the root. The latter is a decision-making module that is responsible for short-term decisions on avoiding obstacles during the near future steps of USV exploitation within the context of the value function. Simulations were performed using two algorithms: a basic vanilla vessel navigator (VVN) as a baseline and an improved one for the vessel navigator with a planner and local view (VNPLV). Experimental results show that the proposed method enhanced the performance of VVN by 55.31 on average for long-distance missions. Our model successfully demonstrated obstacle avoidance by means of deep reinforcement learning using planning adaptive paths in unknown environments.

What problem does this paper attempt to address?

This paper attempts to solve the path planning and obstacle avoidance problems faced by unmanned surface vessels (USVs) during mission execution. Specifically, the author aims to enhance the autonomous navigation ability of unmanned vessels in unknown environments by combining deep reinforcement learning and local view strategies. ### Main problems: 1. **Path planning**: How to generate a safe and efficient path from the starting point to the target point. 2. **Obstacle avoidance**: How to dynamically avoid obstacles in an unknown environment and ensure that the vessel reaches the destination safely. ### Solutions: To address the above challenges, the author proposes a new method - **Vessel Navigator with Planner and Local View (VNPLV)**, which utilizes the following techniques: - **Deep Reinforcement Learning (DRL)**: Through the Deep Q - Learning algorithm, the model can make decisions in complex environments. - **Rolling Wave Planning**: This is a concept in agile methodology, combining long - term planning and short - term decision - making to adapt to the dynamic changes of the environment. - **Path simplification**: Use the Ramer - Douglas - Peucker algorithm to reduce redundant points in the path and improve flexibility. - **Local view strategy**: Only focus on the local area around the vessel, reduce the complexity of the state space, and at the same time avoid the influence of distant static obstacles on decision - making. ### Experimental results: Experiments show that the VNPLV method has a 55.31% performance improvement compared to the basic Vanilla Vessel Navigator (VVN) method in long - distance tasks, and can avoid obstacles more effectively and reach the destination. ### Summary: The main contribution of this paper is to develop a new method that combines deep reinforcement learning and local view strategies for path planning and obstacle avoidance of unmanned surface vessels. This method not only improves the autonomous navigation ability of vessels in unknown environments but also shows the potential of applying deep learning techniques in complex marine environments.

Using Deep Reinforcement Learning Methods for Autonomous Vessels in 2D Environments

Unmanned Surface Vehicle Aided Maritime Data Collection Using Deep Reinforcement Learning

Dynamic Obstacle Avoidance for USVs Using Cross-Domain Deep Reinforcement Learning and Neural Network Model Predictive Controller

AI on the Water: Applying DRL to Autonomous Vessel Navigation

Deep reinforcement learning for adaptive path planning and control of an autonomous underwater vehicle

Obstacle avoidance USV in multi-static obstacle environments based on a deep reinforcement learning approach

Neural Network Model-Based Reinforcement Learning Control for AUV 3-D Path Following

Impact of tight optical filtering on orthogonal time-frequency domain multiplexed signals in wavelength-selective switching systems

Deep Interactive Reinforcement Learning for Path Following of Autonomous Underwater Vehicle

An Algorithm of Complete Coverage Path Planning for Unmanned Surface Vehicle Based on Reinforcement Learning

A novel intelligent collision avoidance algorithm based on deep reinforcement learning approach for USV

Implantable cardioverter defibrillators after acute myocardial infarction

Robust Unmanned Surface Vehicle Navigation with Distributional Reinforcement Learning

Sim-to-Real: Mapless Navigation for USVs Using Deep Reinforcement Learning

2-Level Reinforcement Learning for Ships on Inland Waterways: Path Planning and Following

Path Planning of Unmanned Underwater Vehicles Based on Deep Reinforcement Learning Algorithm

An Autonomous Path Planning Model for Unmanned Ships Based on Deep Reinforcement Learning

A human-like collision avoidance method for USVs based on deep reinforcement learning and velocity obstacle

Path Planning of Maritime Autonomous Surface Ships in Unknown Environment with Reinforcement Learning

A path planning strategy unified with a COLREGS collision avoidance function based on deep reinforcement learning and artificial potential field

Aeolus Ocean -- A simulation environment for the autonomous COLREG-compliant navigation of Unmanned Surface Vehicles using Deep Reinforcement Learning and Maritime Object Detection