Abstract:Reinforcement learning (RL) has achieved remarkable advancements in navigation tasks in recent years. However, tackling multi-goal navigation tasks with sparse rewards remains a complex and challenging problem due to the long-sequence decision-making involved. Such multi-goal navigation tasks inherently incorporate a hybrid action space, where the robot needs to select a navigation endpoint first before executing primitive actions. To address the problem of multi-goal navigation with sparse rewards, we introduce a novel hierarchical RL framework named Hierarchical RL with Multi-Goal (HRL-MG). The main idea of HRL-MG is to divide and conquer the hybrid action space, splitting long-sequence decisions into short-sequence decisions. The HRL-MG framework is composed of two main modules: a selector and an actuator. The selector employs a temporal abstraction hierarchical architecture designed to specify a desired end goal based on the discrete action space. Conversely, the actuator utilizes a continuous goal-oriented hierarchical architecture developed to enact continuous action sequences to reach the desired end goal specified by the selector. In addition, we incorporate a dynamic goal detection mechanism, grounded in hindsight experience replay, to mitigate the challenges posed by sparse reward landscapes. We validated the algorithm's efficacy on both the discrete environment Maze_2D and the continuous robotic environment MuJoCo 'Ant'. The results indicate that HRL-MG significantly outperforms other methods in multi-goal navigation tasks with sparse rewards.

Unbiased Methods for Multi-Goal Reinforcement Learning

Density-based Curriculum for Multi-goal Reinforcement Learning with Sparse Rewards

Generalizing Across Multi-Objective Reward Functions in Deep Reinforcement Learning

Quantile Regression Hindsight Experience Replay

Bias-reduced Multi-step Hindsight Experience Replay for Efficient Multi-goal Reinforcement Learning

Maximum Entropy-Regularized Multi-Goal Reinforcement Learning

Hierarchical reinforcement learning for handling sparse rewards in multi-goal navigation

Efficient Multi-Goal Reinforcement Learning Via Value Consistency Prioritization

Addressing Hindsight Bias in Multigoal Reinforcement Learning

Revisiting Sparse Rewards for Goal-Reaching Reinforcement Learning

Inverse Reinforcement Learning with Multiple Ranked Experts

Generating Attentive Goals for Prioritized Hindsight Reinforcement Learning

MHER: Model-based Hindsight Experience Replay

Hindsight Task Relabelling: Experience Replay for Sparse Reward Meta-RL

Learning Fair Policies in Multi-Objective (deep) Reinforcement Learning with Average and Discounted Rewards.

Learning Representations in Model-Free Hierarchical Reinforcement Learning

Efficient Sparse-Reward Goal-Conditioned Reinforcement Learning with a High Replay Ratio and Regularization

Reinforcement Learning from LLM Feedback to Counteract Goal Misgeneralization

Combining Hindsight with Goal-enhanced Prediction for Multi-goal Reinforcement Learning

Multiobjective Reinforcement Learning: A Comprehensive Overview

Solving Robotic Manipulation With Sparse Reward Reinforcement Learning Via Graph-Based Diversity and Proximity