Robust Unmanned Surface Vehicle Navigation with Distributional Reinforcement Learning

Xi Lin,John McConnell,Brendan Englot

DOI: https://doi.org/10.48550/arXiv.2307.16240

2023-07-30

Abstract:Autonomous navigation of Unmanned Surface Vehicles (USV) in marine environments with current flows is challenging, and few prior works have addressed the sensorbased navigation problem in such environments under no prior knowledge of the current flow and obstacles. We propose a Distributional Reinforcement Learning (RL) based local path planner that learns return distributions which capture the uncertainty of action outcomes, and an adaptive algorithm that automatically tunes the level of sensitivity to the risk in the environment. The proposed planner achieves a more stable learning performance and converges to safer policies than a traditional RL based planner. Computational experiments demonstrate that comparing to a traditional RL based planner and classical local planning methods such as Artificial Potential Fields and the Bug Algorithm, the proposed planner is robust against environmental flows, and is able to plan trajectories that are superior in safety, time and energy consumption.

Robotics

What problem does this paper attempt to address?

This paper attempts to solve the problem of unmanned surface vehicle (USV) sensor navigation in an environment with unknown ocean currents and obstacles. Specifically, the paper focuses on how to achieve local path planning in a complex marine environment through sensor data without prior knowledge. Traditional reinforcement learning (RL) methods have limitations when dealing with such highly uncertain environments. Therefore, this paper proposes a local path planner based on distributional reinforcement learning (Distributional RL), aiming to improve the navigation performance of USV in unknown environments, especially the safety and efficiency when facing ocean current interference and obstacles. The main contributions of the paper include: - Proposing the first local path planner for USV sensor navigation based on distributional reinforcement learning, which is suitable for environments with unknown ocean currents and obstacles. - Verifying through simulation experiments that the performance of the proposed method in terms of safety, time, and energy consumption is superior to that of traditional RL and classical reactive planning algorithms. - Open - sourcing the software implementation of this method and the simulation environment used for researching USV navigation decisions.

Robust Unmanned Surface Vehicle Navigation with Distributional Reinforcement Learning

Distributional Reinforcement Learning based Integrated Decision Making and Control for Autonomous Surface Vehicles

Decentralized Multi-Robot Navigation for Autonomous Surface Vehicles with Distributional Reinforcement Learning

Neural Network Model-Based Reinforcement Learning Control for AUV 3-D Path Following

Using Deep Reinforcement Learning Methods for Autonomous Vessels in 2D Environments

Robust ASV Navigation Through Ground to Water Cross-Domain Deep Reinforcement Learning

Path Planning based on Deep Reinforcement Learning for Autonomous Underwater Vehicles under Ocean Current Disturbance

Robust Route Planning with Distributional Reinforcement Learning in a Stochastic Road Network Environment

Comprehensive Ocean Information-Enabled AUV Motion Planning Based on Reinforcement Learning

Local path planning for unmanned surface vehicle based on spatial and temporal sensing-enhanced deep Q-network

Path Planning of Unmanned Surface Vehicle Based on Improved Q-Learning Algorithm

Path Planning of Unmanned Underwater Vehicles Based on Deep Reinforcement Learning Algorithm

AUV Path Planning with Kinematic Constraints in Unknown Environment Using Reinforcement Learning.

An Offline Reinforcement Learning Approach for Path Following of an Unmanned Surface Vehicle

Path Following Control for Unmanned Surface Vehicles: A Reinforcement Learning-Based Method with Experimental Validation.

Deep Reinforcement Learning-Based Path Planning of Underactuated Surface Vessels

A Real-time Algorithm for USV Navigation Based on Deep Reinforcement Learning

A Path Planning Method Based on Deep Reinforcement Learning for AUV in Complex Marine Environment

Sim-to-Real: Mapless Navigation for USVs Using Deep Reinforcement Learning

An Algorithm of Complete Coverage Path Planning for Unmanned Surface Vehicle Based on Reinforcement Learning

End-to-End Trajectory Tracking Algorithm for Unmanned Surface Vehicle Using Reinforcement Learning