Abstract:Common approaches to Reinforcement Learning (RL) are seriously challenged by large-scale applications involving huge state spaces and sparse delayed reward feedback. Hierarchical Reinforcement Learning (HRL) methods attempt to address this scalability issue by learning action selection policies at multiple levels of temporal abstraction. Abstraction can be had by identifying a relatively small set of states that are likely to be useful as subgoals, in concert with the learning of corresponding skill policies to achieve those subgoals. Many approaches to subgoal discovery in HRL depend on the analysis of a model of the environment, but the need to learn such a model introduces its own problems of scale. Once subgoals are identified, skills may be learned through intrinsic motivation, introducing an internal reward signal marking subgoal attainment. In this paper, we present a novel model-free method for subgoal discovery using incremental unsupervised learning over a small memory of the most recent experiences (trajectories) of the agent. When combined with an intrinsic motivation learning mechanism, this method learns both subgoals and skills, based on experiences in the environment. Thus, we offer an original approach to HRL that does not require the acquisition of a model of the environment, suitable for large-scale applications. We demonstrate the efficiency of our method on two RL problems with sparse delayed feedback: a variant of the rooms environment and the first screen of the ATARI 2600 Montezuma's Revenge game.

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is the scalability challenges faced by traditional Reinforcement Learning (RL) methods in large - scale applications. Specifically: 1. **Large - scale state space and sparse delayed reward feedback**: Traditional RL methods perform poorly when dealing with tasks having a huge state space and sparse delayed reward feedback. These problems make the learning process very inefficient and may even prevent convergence to the optimal policy. 2. **Challenges in sub - goal discovery**: Hierarchical Reinforcement Learning (HRL) addresses these scalability issues by learning action - selection strategies at different levels of temporal abstraction. However, identifying useful sub - goals is a key problem in HRL. Many existing sub - goal discovery methods rely on the analysis of environmental models, and building such models itself also poses scale - related challenges. 3. **Sub - goal discovery without environmental models**: To overcome the above challenges, this paper proposes a new model - free method to automatically discover sub - goals by incrementally performing unsupervised learning on recent experience trajectories. This method does not require explicit learning of environmental models, making it more suitable for large - scale applications. 4. **Skill learning driven by intrinsic motivation**: Once sub - goals are identified, corresponding skills can be learned through the intrinsic motivation mechanism. Intrinsic motivation provides an internal reward signal for achieving sub - goals, thus guiding the agent to learn how to efficiently reach these sub - goals. 5. **Efficient representation learning**: In addition, the paper also explores methods for integrating internal representation learning, temporal abstraction, automatic sub - goal discovery, and intrinsic motivation learning within the model - free HRL framework. These methods aim to automatically extract useful state features from the agent's experience to support generalization ability in large - scale tasks. In summary, this paper mainly solves the following problems: - How to effectively discover sub - goals without relying on environmental models. - How to efficiently learn skills for achieving sub - goals through the intrinsic motivation mechanism. - How to perform effective representation learning in large - scale states to support broader generalization ability. These methods together form a unified model - free HRL algorithm, which is suitable for large - scale reinforcement learning tasks with sparse delayed feedback.

Learning Representations in Model-Free Hierarchical Reinforcement Learning

Efficient Exploration through Intrinsic Motivation Learning for Unsupervised Subgoal Discovery in Model-Free Hierarchical Reinforcement Learning

Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation

Hierarchical reinforcement learning with natural language subgoals

Goal Space Abstraction in Hierarchical Reinforcement Learning via Set-Based Reachability Analysis

Hierarchical Reinforcement Learning: A Survey and Open Research Challenges

Multi-Horizon Representations with Hierarchical Forward Models for Reinforcement Learning

Goal Space Abstraction in Hierarchical Reinforcement Learning via Reachability Analysis

Exploring the limits of Hierarchical World Models in Reinforcement Learning

Feature Control as Intrinsic Motivation for Hierarchical Reinforcement Learning

Int-HRL: Towards Intention-based Hierarchical Reinforcement Learning

Probabilistic Subgoal Representations for Hierarchical Reinforcement learning

Hierarchical Reinforcement Learning in Complex 3D Environments

Active Hierarchical Exploration with Stable Subgoal Representation Learning

Hierarchical Reinforcement Learning with Timed Subgoals

Screening Goals and Selecting Policies in Hierarchical Reinforcement Learning.

Summarize of hierarchical reinforcement learning

HCS-R-HER: Hierarchical Reinforcement Learning Based on Cross Subtasks Rainbow Hindsight Experience Replay

Hierarchical reinforcement learning for handling sparse rewards in multi-goal navigation

Deep Reinforcement Learning with Hierarchical Structures.

Subgoal-based Hierarchical Reinforcement Learning for Multi-Agent Collaboration