Abstract:A hallmark of human intelligence, but challenging for reinforcement learning (RL) agents, is the ability to compositionally generalise, that is, to recompose familiar knowledge components in novel ways to solve new problems. For instance, when navigating in a city, one needs to know the location of the destination and how to operate a vehicle to get there, whether it be pedalling a bike or operating a car. In RL, these correspond to the reward function and transition function, respectively. To compositionally generalize, these two components need to be transferable independently of each other: multiple modes of transport can reach the same goal, and any given mode can be used to reach multiple destinations. Yet there are also instances where it can be helpful to learn and transfer entire structures, jointly representing goals and transitions, particularly whenever these recur in natural tasks (e.g., given a suggestion to get ice cream, one might prefer to bike, even in new towns). Prior theoretical work has explored how, in model-based RL, agents can learn and generalize task components (transition and reward functions). But a satisfactory account for how a single agent can simultaneously satisfy the two competing demands is still lacking. Here, we propose a hierarchical RL agent that learns and transfers individual task components as well as entire structures (particular compositions of components) by inferring both through a non-parametric Bayesian model of the task. It maintains a factorised representation of task components through a hierarchical Dirichlet process, but it also represents different possible covariances between these components through a standard Dirichlet process. We validate our approach on a variety of navigation tasks covering a wide range of statistical correlations between task components and show that it can also improve generalisation and transfer in more complex, hierarchical tasks with goal/subgoal structures. Finally, we end with a discussion of our work including how this clustering algorithm could conceivably be implemented by cortico-striatal gating circuits in the brain.

Computational evidence for hierarchically structured reinforcement learning in humans

Hierarchically organized behavior and its neural foundations: A reinforcement learning perspective

Hierarchical reinforcement learning and decision making

Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?

Exploring the limits of Hierarchical World Models in Reinforcement Learning

An algorithmic account for how humans efficiently learn, transfer, and compose hierarchically structured decision policies

Mechanisms of Hierarchical Reinforcement Learning in Corticostriatal Circuits 1: Computational Analysis

Deep Reinforcement Learning with Hierarchical Structures.

Hierarchical clustering optimizes the tradeoff between compositionality and expressivity of task structures for flexible reinforcement learning

Hierarchical principles of embodied reinforcement learning: A review

Hierarchical Reinforcement Learning: A Survey and Open Research Challenges

Hierarchical reinforcement learning with natural language subgoals

HTMRL: Biologically Plausible Reinforcement Learning with Hierarchical Temporal Memory

Hierarchical Reinforcement Learning in Complex 3D Environments

Learning Representations in Model-Free Hierarchical Reinforcement Learning

On Credit Assignment in Hierarchical Reinforcement Learning

Human Learning of Hierarchical Graphs

Multi-Horizon Representations with Hierarchical Forward Models for Reinforcement Learning

Summarize of hierarchical reinforcement learning

Hierarchical Reinforcement Learning, Sequential Behavior, and the Dorsal Frontostriatal System