Abstract:Reinforcement Learning (RL) is a promising solution, allowing Unmanned Underwater Vehicles (UUVs) to learn optimal behaviors through trial and error. However, existing simulators lack efficient integration with RL methods, limiting training scalability and performance. This paper introduces MarineGym, a novel simulation framework designed to enhance RL training efficiency for UUVs by utilizing GPU acceleration. MarineGym offers a 10,000-fold performance improvement over real-time simulation on a single GPU, enabling rapid training of RL algorithms across multiple underwater tasks. Key features include realistic dynamic modeling of UUVs, parallel environment execution, and compatibility with popular RL frameworks like PyTorch and TorchRL. The framework is validated through four distinct tasks: station-keeping, circle tracking, helical tracking, and lemniscate tracking. This framework sets the stage for advancing RL in underwater robotics and facilitating efficient training in complex, dynamic environments.

What problem does this paper attempt to address?

This paper aims to address the challenges of achieving autonomous control of unmanned underwater vehicles (UUVs) in complex and variable underwater environments. Specifically, the paper focuses on the following issues: 1. **Non - linear Dynamics**: The dynamic characteristics of UUVs are non - linear, which makes it very difficult for them to achieve autonomous control in changing underwater environments. 2. **Challenges in the Application of Reinforcement Learning (RL)**: Although reinforcement learning methods can theoretically improve the autonomous capabilities of UUVs, there are many challenges in practical applications, such as complex environments, high risks, and long training times. 3. **Limitations of Existing Simulators**: Current underwater robot simulators have deficiencies in terms of compatibility with reinforcement learning methods and lack of attention to training efficiency. To solve these problems, the paper introduces a new UUV simulation framework - MarineGym. This framework is developed based on Isaac Sim, and its main contributions include: - **Efficient Training**: Through GPU acceleration, MarineGym achieves a 10,000 - fold performance improvement over real - time, reducing the training time to a few minutes. - **Accurate Replication of the Physical Environment**: It can accurately simulate physical laws, kinematics, and dynamics. - **Parallel Execution**: It supports the parallel execution of multiple environment instances, improving training efficiency and sample diversity. - **Compatibility with Existing RL Frameworks**: It is compatible with existing reinforcement learning frameworks (such as TorchRL) and provides a user - friendly API. The paper also designs four different control tasks to verify the effectiveness of MarineGym, including fixed - point holding, circular trajectory tracking, spiral trajectory tracking, and figure - eight trajectory tracking. The experimental results show that this framework can complete training in a short time and performs excellently in various tasks. For example, the position error in the fixed - point holding task is less than 0.1 meters, and the error in the trajectory tracking task does not exceed 0.02 meters.

MarineGym: Accelerated Training for Underwater Vehicles with High-Fidelity RL Simulation

Learning an End-To-End Policy for AUV Control Within Just Forty Minutes Using Parallel Simulation

UUVSim: Intelligent Modular Simulation Platform for Unmanned Underwater Vehicle Learning

FishGym: A High-Performance Physics-based Simulation Framework for Underwater Robot Learning

Underwater Multi-agent Cooperative Formation Hunting Based on Deep Reinforcement Learning

Deep Interactive Reinforcement Learning for Path Following of Autonomous Underwater Vehicle

Dynamic Obstacle Avoidance for USVs Using Cross-Domain Deep Reinforcement Learning and Neural Network Model Predictive Controller

Aquatic Navigation: A Challenging Benchmark for Deep Reinforcement Learning

Multi-AUV Pursuit-Evasion Game in the Internet of Underwater Things: an Efficient Training Framework Via Offline Reinforcement Learning

Underwater Target Tracking Based on Hierarchical Software-Defined Multi-AUV Reinforcement Learning: A Multi-AUV Advantage-Attention Actor-Critic Approach

A Simulator and First Reinforcement Learning Results for Underwater Mapping

Robust ASV Navigation Through Ground to Water Cross-Domain Deep Reinforcement Learning

A Real-time Algorithm for USV Navigation Based on Deep Reinforcement Learning

Learning Agile Swimming: An End-to-End Approach without CPGs

Enhancing Underwater IoT Security: A Collaborative Pursuit Strategy Using Multi-Agent Reinforcement Learning

Asynchronous Multithreading Reinforcement-Learning-Based Path Planning and Tracking for Unmanned Underwater Vehicle

Multi-Objective-Optimization Multi-AUV Assisted Data Collection Framework for IoUT Based on Offline Reinforcement Learning

UNav-Sim: A Visually Realistic Underwater Robotics Simulator and Synthetic Data-generation Framework

Continuous Control for Autonomous Underwater Vehicle Path Following Using Deep Interactive Reinforcement Learning

AutoVRL: A High Fidelity Autonomous Ground Vehicle Simulator for Sim-to-Real Deep Reinforcement Learning

Target Tracking Control of a Biomimetic Underwater Vehicle Through Deep Reinforcement Learning