Abstract:Vision-and-language navigation requires an agent to navigate in a photo-realistic environment by following natural language instructions. Mainstream methods employ imitation learning (IL) to let the agent imitate the behavior of the teacher. The trained model will overfit the teacher's biased behavior, resulting in poor model generalization. Recently, researchers have sought to combine IL and reinforcement learning (RL) to overcome overfitting and enhance model generalization. However, these methods still face the problem of expensive trajectory annotation. We propose a hierarchical RL-based method-discovering intrinsic subgoals via hierarchical (DISH) RL-which overcomes the generalization limitations of current methods and gets rid of expensive label annotations. First, the high-level agent (manager) decomposes the complex navigation problem into simple intrinsic subgoals. Then, the low-level agent (worker) uses an intrinsic subgoal-driven attention mechanism for action prediction in a smaller state space. We place no constraints on the semantics that subgoals may convey, allowing the agent to autonomously learn intrinsic, more generalizable subgoals from navigation tasks. Furthermore, we design a novel history-aware discriminator (HAD) for the worker. The discriminator incorporates historical information into subgoal discrimination and provides the worker with additional intrinsic rewards to alleviate the reward sparsity. Without labeled actions, our method provides supervision for the worker in the form of self-supervision by generating subgoals from the manager. The final results of multiple comparison experiments on the Room-to-Room (R2R) dataset show that our DISH can significantly outperform the baseline in accuracy and efficiency.

Hierarchical Reinforcement Learning with Automatic Sub-Goal Identification

HILONet: Hierarchical Imitation Learning from Non-Aligned Observations

Learning Hierarchical Graph-Based Policy for Goal-Reaching in Unknown Environments

Hierarchical reinforcement learning with natural language subgoals

HCS-R-HER: Hierarchical Reinforcement Learning Based on Cross Subtasks Rainbow Hindsight Experience Replay

Subgoal-based Hierarchical Reinforcement Learning for Multi-Agent Collaboration

Efficient Hierarchical Exploration with an Active Subgoal Generation Strategy.

Discovering Intrinsic Subgoals for Vision-and-Language Navigation via Hierarchical Reinforcement Learning

Hierarchical Reinforcement Learning with Attention Reward

Hierarchical reinforcement learning for handling sparse rewards in multi-goal navigation

Active Hierarchical Exploration with Stable Subgoal Representation Learning

Searching Latent Sub-Goals in Hierarchical Reinforcement Learning as Riemannian Manifold Optimization

Connect-Based Subgoal Discovery for Options in Hierarchical Reinforcement Learning

Efficient Exploration through Intrinsic Motivation Learning for Unsupervised Subgoal Discovery in Model-Free Hierarchical Reinforcement Learning

Balancing Exploration and Exploitation in Hierarchical Reinforcement Learning via Latent Landmark Graphs

Goal Space Abstraction in Hierarchical Reinforcement Learning via Reachability Analysis

Learning Representations in Model-Free Hierarchical Reinforcement Learning

Feature Control as Intrinsic Motivation for Hierarchical Reinforcement Learning

Robot Subgoal-guided Navigation in Dynamic Crowded Environments with Hierarchical Deep Reinforcement Learning

Generating Adjacency-Constrained Subgoals in Hierarchical Reinforcement Learning

Hierarchical automatic curriculum learning: Converting a sparse reward navigation task into dense reward.