Abstract:Hierarchical model-based reinforcement learning (HMBRL) aims to combine the benefits of better sample efficiency of model based reinforcement learning (MBRL) with the abstraction capability of hierarchical reinforcement learning (HRL) to solve complex tasks efficiently. While HMBRL has great potential, it still lacks wide adoption. In this work we describe a novel HMBRL framework and evaluate it thoroughly. To complement the multi-layered decision making idiom characteristic for HRL, we construct hierarchical world models that simulate environment dynamics at various levels of temporal abstraction. These models are used to train a stack of agents that communicate in a top-down manner by proposing goals to their subordinate agents. A significant focus of this study is the exploration of a static and environment agnostic temporal abstraction, which allows concurrent training of models and agents throughout the hierarchy. Unlike most goal-conditioned H(MB)RL approaches, it also leads to comparatively low dimensional abstract actions. Although our HMBRL approach did not outperform traditional methods in terms of final episode returns, it successfully facilitated decision making across two levels of abstraction using compact, low dimensional abstract actions. A central challenge in enhancing our method's performance, as uncovered through comprehensive experimentation, is model exploitation on the abstract level of our world model stack. We provide an in depth examination of this issue, discussing its implications for the field and suggesting directions for future research to overcome this challenge. By sharing these findings, we aim to contribute to the broader discourse on refining HMBRL methodologies and to assist in the development of more effective autonomous learning systems for complex decision-making environments.

Efficient Exploration through Intrinsic Motivation Learning for Unsupervised Subgoal Discovery in Model-Free Hierarchical Reinforcement Learning

Efficient Hierarchical Exploration with an Active Subgoal Generation Strategy.

Discovering Intrinsic Subgoals for Vision-and-Language Navigation via Hierarchical Reinforcement Learning

Goal Space Abstraction in Hierarchical Reinforcement Learning via Reachability Analysis

Active Hierarchical Exploration with Stable Subgoal Representation Learning

Hierarchical reinforcement learning with natural language subgoals

Searching Latent Sub-Goals in Hierarchical Reinforcement Learning as Riemannian Manifold Optimization

Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation

Feature Control as Intrinsic Motivation for Hierarchical Reinforcement Learning

Subgoal-based Hierarchical Reinforcement Learning for Multi-Agent Collaboration

Causality-driven Hierarchical Structure Discovery for Reinforcement Learning

Goal Space Abstraction in Hierarchical Reinforcement Learning via Set-Based Reachability Analysis

Connect-Based Subgoal Discovery for Options in Hierarchical Reinforcement Learning

Balancing Exploration and Exploitation in Hierarchical Reinforcement Learning via Latent Landmark Graphs

An agent with a sense of direction for option discovery in hierarchical reinforcement learning

Hierarchical Reinforcement Learning: A Survey and Open Research Challenges

Hierarchical reinforcement learning for handling sparse rewards in multi-goal navigation

Exploring the limits of Hierarchical World Models in Reinforcement Learning

HAC Explore: Accelerating Exploration with Hierarchical Reinforcement Learning

Unsupervised Learning of Goal Spaces for Intrinsically Motivated Goal Exploration

Learning Goal Embeddings via Self-Play for Hierarchical Reinforcement Learning