Abstract:Hierarchical model-based reinforcement learning (HMBRL) aims to combine the benefits of better sample efficiency of model based reinforcement learning (MBRL) with the abstraction capability of hierarchical reinforcement learning (HRL) to solve complex tasks efficiently. While HMBRL has great potential, it still lacks wide adoption. In this work we describe a novel HMBRL framework and evaluate it thoroughly. To complement the multi-layered decision making idiom characteristic for HRL, we construct hierarchical world models that simulate environment dynamics at various levels of temporal abstraction. These models are used to train a stack of agents that communicate in a top-down manner by proposing goals to their subordinate agents. A significant focus of this study is the exploration of a static and environment agnostic temporal abstraction, which allows concurrent training of models and agents throughout the hierarchy. Unlike most goal-conditioned H(MB)RL approaches, it also leads to comparatively low dimensional abstract actions. Although our HMBRL approach did not outperform traditional methods in terms of final episode returns, it successfully facilitated decision making across two levels of abstraction using compact, low dimensional abstract actions. A central challenge in enhancing our method's performance, as uncovered through comprehensive experimentation, is model exploitation on the abstract level of our world model stack. We provide an in depth examination of this issue, discussing its implications for the field and suggesting directions for future research to overcome this challenge. By sharing these findings, we aim to contribute to the broader discourse on refining HMBRL methodologies and to assist in the development of more effective autonomous learning systems for complex decision-making environments.

Specialization in Hierarchical Learning Systems

An Information-theoretic On-line Learning Principle for Specialization in Hierarchical Decision-Making Systems

Multi-group Learning for Hierarchical Groups

Hierarchical Learning Algorithms for Multi-scale Expert Problems

Expertise Trees Resolve Knowledge Limitations in Collective Decision-Making

Decision-Making by Hierarchies of Discordant Agents

An algorithmic account for how humans efficiently learn, transfer, and compose hierarchically structured decision policies

On The Specialization of Neural Modules

Hierarchical reinforcement learning and decision making

Exploring the limits of Hierarchical World Models in Reinforcement Learning

Automatic Discovery and Transfer of Maxq Hierarchies in A Complex System

A Hierarchical Framework for Cooperative Tasks in Multi-agent Systems

Heuristic-Based Weak Learning for Automated Decision-Making

Hierarchical Policy Learning is Sensitive to Goal Space Design

Optimal Hierarchical Learning Path Design with Reinforcement Learning

Hierarchical Lifelong Learning by Sharing Representations and Integrating Hypothesis.

Hierarchical Reinforcement Learning Based Multi-Agent Collaborative Control Approach

Hierarchical Method for Cooperative Multiagent Reinforcement Learning in Markov Decision Processes

Hierarchical Subtask Discovery With Non-Negative Matrix Factorization

Evolving hierarchical memory-prediction machines in multi-task reinforcement learning

Large-scale Group Hierarchical DEMATEL Method with Automatic Consensus Reaching