Abstract:Reinforcement learning (RL) has achieved remarkable advancements in navigation tasks in recent years. However, tackling multi-goal navigation tasks with sparse rewards remains a complex and challenging problem due to the long-sequence decision-making involved. Such multi-goal navigation tasks inherently incorporate a hybrid action space, where the robot needs to select a navigation endpoint first before executing primitive actions. To address the problem of multi-goal navigation with sparse rewards, we introduce a novel hierarchical RL framework named Hierarchical RL with Multi-Goal (HRL-MG). The main idea of HRL-MG is to divide and conquer the hybrid action space, splitting long-sequence decisions into short-sequence decisions. The HRL-MG framework is composed of two main modules: a selector and an actuator. The selector employs a temporal abstraction hierarchical architecture designed to specify a desired end goal based on the discrete action space. Conversely, the actuator utilizes a continuous goal-oriented hierarchical architecture developed to enact continuous action sequences to reach the desired end goal specified by the selector. In addition, we incorporate a dynamic goal detection mechanism, grounded in hindsight experience replay, to mitigate the challenges posed by sparse reward landscapes. We validated the algorithm's efficacy on both the discrete environment Maze_2D and the continuous robotic environment MuJoCo 'Ant'. The results indicate that HRL-MG significantly outperforms other methods in multi-goal navigation tasks with sparse rewards.

Hierarchical automatic curriculum learning: Converting a sparse reward navigation task into dense reward.

Hierarchical reinforcement learning for handling sparse rewards in multi-goal navigation

HCS-R-HER: Hierarchical Reinforcement Learning Based on Cross Subtasks Rainbow Hindsight Experience Replay

Variational Automatic Curriculum Learning for Sparse-Reward Cooperative Multi-Agent Problems

Hierarchical Multi-Agent Reinforcement Learning for Cooperative Tasks with Sparse Rewards in Continuous Domain

HAC Explore: Accelerating Exploration with Hierarchical Reinforcement Learning

Hierarchical Reinforcement Learning with Automatic Sub-Goal Identification

Hierarchical reinforcement learning for efficient exploration and transfer

Learning Hierarchical Behavior and Motion Planning for Autonomous Driving.

Curriculum-RL Based Air Combat Decision-Making

Hierarchical Reinforcement Learning with Attention Reward

Towards Skilled Population Curriculum for Multi-Agent Reinforcement Learning

Hierarchical Curriculum Learning for AMR Parsing

HG2P: Hippocampus-inspired High-reward Graph and Model-Free Q-Gradient Penalty for Path Planning and Motion Control

Feature Control as Intrinsic Motivation for Hierarchical Reinforcement Learning

Subgoal-based Hierarchical Reinforcement Learning for Multi-Agent Collaboration

Automated Curriculum Reinforcement Learning in Path-finding Scenarios

Automatic Curriculum Learning with Gradient Reward Signals

Density-based Curriculum for Multi-goal Reinforcement Learning with Sparse Rewards

On Credit Assignment in Hierarchical Reinforcement Learning

Hierarchical Reinforcement Learning with Advantage-Based Auxiliary Rewards