Abstract:Conventional hierarchical reinforcement learning (HRL) relies on discrete options to represent explicitly distinguishable knowledge, which may lead to severe performance bottlenecks. It is possible to represent richer knowledge through continuous options, but reliable scheduling methods are lacking. To design an available scheduling method for continuous options, in this paper, the hierarchical reinforcement learning with adaptive scheduling (HAS) algorithm is proposed. Its low-level controller learns diverse options, while the high-level controller schedules options to learn solutions. It achieves an adaptive balance between exploration and exploitation during the frequent scheduling of continuous options, maximizing the representation potential of continuous options. It builds on multi-step static scheduling and makes switching decisions according to the relative advantages of the previous and the estimated continuous options, enabling the agent to focus on different behaviors at different phases of the task. The expected t-step distance is applied to demonstrate the superiority of adaptive scheduling in terms of exploration. Furthermore, an interruption incentive based on annealing is proposed to alleviate excessive exploration during the early training phase, accelerating the convergence rate. Finally, we apply HAS to robot control with sparse rewards in continuous spaces, and develop a comprehensive experimental analysis scheme. The experimental results not only demonstrate the high performance and robustness of HAS, but also provide evidence that the adaptive scheduling method has a positive effect both on the representation and option policies.

Hierarchical Reinforcement Learning for Kinematic Control Tasks with Parameterized Action Spaces

Demonstration Data-Driven Parameter Adjustment for Trajectory Planning in Highly Constrained Environments

Learning Hierarchical Behavior and Motion Planning for Autonomous Driving.

Learning with Training Wheels: Speeding up Training with a Simple Controller for Deep Reinforcement Learning

Hierarchical Deep Reinforcement Learning for Continuous Action Control

Deep Multi-Agent Reinforcement Learning with Discrete-Continuous Hybrid Action Spaces

Data-Efficient Hierarchical Reinforcement Learning for Robotic Assembly Control Applications

Hybrid Actor-Critic Reinforcement Learning in Parameterized Action Space

Model-based Reinforcement Learning for Parameterized Action Spaces

Reinforcement Learning with Function-Valued Action Spaces for Partial Differential Equation Control

Action and Trajectory Planning for Urban Autonomous Driving with Hierarchical Reinforcement Learning

ReACT: Reinforcement Learning for Controller Parametrization using B-Spline Geometries

Hierarchical Intermittent Motor Control with Deterministic Policy Gradient.

Goal-Conditioned Hierarchical Reinforcement Learning with High-Level Model Approximation.

NEARL: Non-Explicit Action Reinforcement Learning for Robotic Control

HyAR: Addressing Discrete-Continuous Action Reinforcement Learning via Hybrid Action Representation

From proprioception to long-horizon planning in novel environments: A hierarchical RL model

Hierarchical Multi-Agent Reinforcement Learning for Cooperative Tasks with Sparse Rewards in Continuous Domain

Hierarchical Reinforcement Learning Based on Planning Operators

Hierarchical Reinforcement Learning with Adaptive Scheduling for Robot Control.

Planning-Augmented Hierarchical Reinforcement Learning