Abstract:The fundamental concept behind option-based hierarchical reinforcement learning (O-HRL) is to obtain temporal coarse-grained actions and abstract complex situations. Although O-HRL is intended for sparse rewards, it remains difficult to extend it to sparse reward problems in continuous spaces. In this paper, we provide a fresh perspective on option technology to comprehend different options based on knowledge representation. The hierarchical reinforcement learning with the unlimited option scheduling (UOS) algorithm is proposed. Unlike conventional O-HRL algorithms that apply a limited set of options with specific meanings, UOS encourages an infinite number of options to correlate with trajectories while maintaining a correlation with each other, thus representing more abundant knowledge. These unlimited options can guide infinite and diverse trajectories to cover fine-grained state spaces. Further, a composite scheduling mode is proposed to generate arbitrary-length trajectories with intrinsic characteristics, providing both flexibility and concentration for unlimited options. It significantly improves the performance and robustness of UOS. Finally, a new comprehensive experimental system is developed, and the experimental results demonstrate the notable success of UOS on sparse reward tasks in continuous spaces. It also identifies the root cause of UOS superiority from the perspective of knowledge representation.

New Method of Hierarchical Reinforcement Learning

Hierarchical reinforcement learning with unlimited option scheduling for sparse rewards in continuous spaces

Autonomous Discovery and Creation of Options in Hierarchical Reinforcement Learning

Option-Based Hierarchical Reinforcement Learning for UAV Multi-Objective Path Planning

Multi-Level Discovery of Deep Options

Online Baum-Welch algorithm for Hierarchical Imitation Learning

Hierarchical Reinforcement Learning via Advantage-Weighted Information Maximization

Optimal Exploration Algorithm of Multi-Agent Reinforcement Learning Methods (Student Abstract)

An agent with a sense of direction for option discovery in hierarchical reinforcement learning

Adversarial Option-Aware Hierarchical Imitation Learning.

POAQL: A Partially Observable Altruistic Q-Learning Method for Cooperative Multi-Agent Reinforcement Learning

Mitigating Relative Over-Generalization in Multi-Agent Reinforcement Learning

Learning to Interrupt: A Hierarchical Deep Reinforcement Learning Framework for Efficient Exploration

A Provably Efficient Option-Based Algorithm for both High-Level and Low-Level Learning

Hierarchical Reinforcement Learning with Advantage-Based Auxiliary Rewards

Hierarchical Reinforcement Learning in Complex 3D Environments

MQES: Max-Q Entropy Search for Efficient Exploration in Continuous Reinforcement Learning

Deep Hierarchical Reinforcement Learning Algorithm in Partially Observable Markov Decision Processes

Algorithms for Batch Hierarchical Reinforcement Learning

HAVEN: Hierarchical Cooperative Multi-Agent Reinforcement Learning with Dual Coordination Mechanism

A Hierarchical Reinforcement Learning Framework for Stock Selection and Portfolio