Abstract:Hierarchical model-based reinforcement learning (HMBRL) aims to combine the benefits of better sample efficiency of model based reinforcement learning (MBRL) with the abstraction capability of hierarchical reinforcement learning (HRL) to solve complex tasks efficiently. While HMBRL has great potential, it still lacks wide adoption. In this work we describe a novel HMBRL framework and evaluate it thoroughly. To complement the multi-layered decision making idiom characteristic for HRL, we construct hierarchical world models that simulate environment dynamics at various levels of temporal abstraction. These models are used to train a stack of agents that communicate in a top-down manner by proposing goals to their subordinate agents. A significant focus of this study is the exploration of a static and environment agnostic temporal abstraction, which allows concurrent training of models and agents throughout the hierarchy. Unlike most goal-conditioned H(MB)RL approaches, it also leads to comparatively low dimensional abstract actions. Although our HMBRL approach did not outperform traditional methods in terms of final episode returns, it successfully facilitated decision making across two levels of abstraction using compact, low dimensional abstract actions. A central challenge in enhancing our method's performance, as uncovered through comprehensive experimentation, is model exploitation on the abstract level of our world model stack. We provide an in depth examination of this issue, discussing its implications for the field and suggesting directions for future research to overcome this challenge. By sharing these findings, we aim to contribute to the broader discourse on refining HMBRL methodologies and to assist in the development of more effective autonomous learning systems for complex decision-making environments.

Learning World Models With Hierarchical Temporal Abstractions: A Probabilistic Perspective

Multi Time Scale World Models

Predictive World Models from Real-World Partial Observations

World model learning and inference

Discovering Latent States for Model Learning: Applying Sensorimotor Contingencies Theory and Predictive Processing to Model Context

Exploring the limits of Hierarchical World Models in Reinforcement Learning

Humans rationally balance detailed and temporally abstract world models

Learning a World Model With Multitimescale Memory Augmentation

A Hierarchical Bayesian Model for Inferring and Decision Making in Multi-Dimensional Volatile Binary Environments

Abstraction-Refinement for Hierarchical Probabilistic Models

Causal World Models by Unsupervised Deconfounding of Physical Dynamics

Episodic Memory for Learning Subjective-Timescale Models

Neural World Models for Computer Vision

Learning in Hybrid Active Inference Models

Hybrid Recurrent Models Support Emergent Descriptions for Hierarchical Planning and Control

HiPPO-Prophecy: State-Space Models can Provably Learn Dynamical Systems in Context

Inferring Time-Varying Internal Models of Agents Through Dynamic Structure Learning

Active Predictive Coding: A Unified Neural Framework for Learning Hierarchical World Models for Perception and Planning

Learning Latent Dynamic Robust Representations for World Models

Learning Discrete State Abstractions With Deep Variational Inference

Learning Dynamics Model in Reinforcement Learning by Incorporating the Long Term Future