Abstract:A hallmark of human intelligence, but challenging for reinforcement learning (RL) agents, is the ability to compositionally generalise, that is, to recompose familiar knowledge components in novel ways to solve new problems. For instance, when navigating in a city, one needs to know the location of the destination and how to operate a vehicle to get there, whether it be pedalling a bike or operating a car. In RL, these correspond to the reward function and transition function, respectively. To compositionally generalize, these two components need to be transferable independently of each other: multiple modes of transport can reach the same goal, and any given mode can be used to reach multiple destinations. Yet there are also instances where it can be helpful to learn and transfer entire structures, jointly representing goals and transitions, particularly whenever these recur in natural tasks (e.g., given a suggestion to get ice cream, one might prefer to bike, even in new towns). Prior theoretical work has explored how, in model-based RL, agents can learn and generalize task components (transition and reward functions). But a satisfactory account for how a single agent can simultaneously satisfy the two competing demands is still lacking. Here, we propose a hierarchical RL agent that learns and transfers individual task components as well as entire structures (particular compositions of components) by inferring both through a non-parametric Bayesian model of the task. It maintains a factorised representation of task components through a hierarchical Dirichlet process, but it also represents different possible covariances between these components through a standard Dirichlet process. We validate our approach on a variety of navigation tasks covering a wide range of statistical correlations between task components and show that it can also improve generalisation and transfer in more complex, hierarchical tasks with goal/subgoal structures. Finally, we end with a discussion of our work including how this clustering algorithm could conceivably be implemented by cortico-striatal gating circuits in the brain.

Generalization of Compositional Tasks with Logical Specification via Implicit Planning

Robust Subtask Learning for Compositional Generalization

Compositional planning in Markov decision processes: Temporal abstraction meets generalized logic composition

Zero-Shot Policy Transfer with Disentangled Task Representation of Meta-Reinforcement Learning.

Generalizing to New Tasks via One-Shot Compositional Subgoals

Discrete Compositional Representations as an Abstraction for Goal Conditioned Reinforcement Learning

Safety-Aware Task Composition for Discrete and Continuous Reinforcement Learning

Hierarchical Programmatic Reinforcement Learning via Learning to Compose Programs

Learning Parameterized Task Structure for Generalization to Unseen Entities

Compositional Generalization by Learning Analytical Expressions.

Hierarchical Reinforcement Learning for Concurrent Discovery of Compound and Composable Policies

Leveraging the Efficiency of Multi-Task Robot Manipulation Via Task-Evoked Planner and Reinforcement Learning

Planning with a Learned Policy Basis to Optimally Solve Complex Tasks

Compositional Policy Learning in Stochastic Control Systems with Formal Guarantees

Inductive Generalization in Reinforcement Learning from Specifications

Verifiable Reinforcement Learning Systems via Compositionality

Refactoring Policy for Compositional Generalizability using Self-Supervised Object Proposals

Hierarchical clustering optimizes the tradeoff between compositionality and expressivity of task structures for flexible reinforcement learning

Synthesizing Programmatic Policy for Generalization Within Task Domain

Towards Task-Prioritized Policy Composition

Categorical semantics of compositional reinforcement learning