Abstract:Conventional hierarchical reinforcement learning (HRL) relies on discrete options to represent explicitly distinguishable knowledge, which may lead to severe performance bottlenecks. It is possible to represent richer knowledge through continuous options, but reliable scheduling methods are lacking. To design an available scheduling method for continuous options, in this paper, the hierarchical reinforcement learning with adaptive scheduling (HAS) algorithm is proposed. Its low-level controller learns diverse options, while the high-level controller schedules options to learn solutions. It achieves an adaptive balance between exploration and exploitation during the frequent scheduling of continuous options, maximizing the representation potential of continuous options. It builds on multi-step static scheduling and makes switching decisions according to the relative advantages of the previous and the estimated continuous options, enabling the agent to focus on different behaviors at different phases of the task. The expected t-step distance is applied to demonstrate the superiority of adaptive scheduling in terms of exploration. Furthermore, an interruption incentive based on annealing is proposed to alleviate excessive exploration during the early training phase, accelerating the convergence rate. Finally, we apply HAS to robot control with sparse rewards in continuous spaces, and develop a comprehensive experimental analysis scheme. The experimental results not only demonstrate the high performance and robustness of HAS, but also provide evidence that the adaptive scheduling method has a positive effect both on the representation and option policies.

Hierarchical Reinforcement Learning with Attention Reward

Hierarchical Reinforcement Learning with Advantage-Based Auxiliary Rewards

Feature Control as Intrinsic Motivation for Hierarchical Reinforcement Learning

Advances in Hierarchical Reinforcement Learning

Deep Reinforcement Learning from Hierarchical Preference Design

Hierarchical Reinforcement Learning: A Survey and Open Research Challenges

Summarize of hierarchical reinforcement learning

HCS-R-HER: Hierarchical Reinforcement Learning Based on Cross Subtasks Rainbow Hindsight Experience Replay

Goal-Conditioned Hierarchical Reinforcement Learning with High-Level Model Approximation.

Hierarchical Reinforcement Learning with Automatic Sub-Goal Identification

Temporal-adaptive Hierarchical Reinforcement Learning

Data-Efficient Hierarchical Reinforcement Learning for Robotic Assembly Control Applications

Balancing Exploration and Exploitation in Hierarchical Reinforcement Learning via Latent Landmark Graphs

Hierarchical Reinforcement Learning with Adaptive Scheduling for Robot Control.

Deep Reinforcement Learning with Hierarchical Structures.

Boosting Reinforcement Learning via Hierarchical Game Playing With State Relay

Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation

Int-HRL: Towards Intention-based Hierarchical Reinforcement Learning

Subgoal-based Hierarchical Reinforcement Learning for Multi-Agent Collaboration

Hierarchical reinforcement learning with natural language subgoals