Abstract:Common approaches to Reinforcement Learning (RL) are seriously challenged by large-scale applications involving huge state spaces and sparse delayed reward feedback. Hierarchical Reinforcement Learning (HRL) methods attempt to address this scalability issue by learning action selection policies at multiple levels of temporal abstraction. Abstraction can be had by identifying a relatively small set of states that are likely to be useful as subgoals, in concert with the learning of corresponding skill policies to achieve those subgoals. Many approaches to subgoal discovery in HRL depend on the analysis of a model of the environment, but the need to learn such a model introduces its own problems of scale. Once subgoals are identified, skills may be learned through intrinsic motivation, introducing an internal reward signal marking subgoal attainment. In this paper, we present a novel model-free method for subgoal discovery using incremental unsupervised learning over a small memory of the most recent experiences (trajectories) of the agent. When combined with an intrinsic motivation learning mechanism, this method learns both subgoals and skills, based on experiences in the environment. Thus, we offer an original approach to HRL that does not require the acquisition of a model of the environment, suitable for large-scale applications. We demonstrate the efficiency of our method on two RL problems with sparse delayed feedback: a variant of the rooms environment and the first screen of the ATARI 2600 Montezuma's Revenge game.

Hierarchical Policy Learning is Sensitive to Goal Space Design

Learning Hierarchical Graph-Based Policy for Goal-Reaching in Unknown Environments

Hierarchical Reinforcement Learning Based on Continuous Subgoal Space

Hierarchical Subspaces of Policies for Continual Offline Reinforcement Learning

Goal Space Abstraction in Hierarchical Reinforcement Learning via Reachability Analysis

Optimal Hierarchical Learning Path Design with Reinforcement Learning

Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?

Goal Space Abstraction in Hierarchical Reinforcement Learning via Set-Based Reachability Analysis

Sub-policy Adaptation for Hierarchical Reinforcement Learning

On the benefits of pixel-based hierarchical policies for task generalization

Temporal-adaptive Hierarchical Reinforcement Learning

Generating Adjacency-Constrained Subgoals in Hierarchical Reinforcement Learning

Probabilistic Subgoal Representations for Hierarchical Reinforcement learning

Hierarchical reinforcement learning with natural language subgoals

Spatially and Seamlessly Hierarchical Reinforcement Learning for State Space and Policy space in Autonomous Driving

Learning Representations in Model-Free Hierarchical Reinforcement Learning

Hierarchical Preference Optimization: Learning to achieve goals via feasible subgoals prediction

Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation

Screening Goals and Selecting Policies in Hierarchical Reinforcement Learning.

Multi-Horizon Representations with Hierarchical Forward Models for Reinforcement Learning

Adjacency Constraint for Efficient Hierarchical Reinforcement Learning