Abstract:A hallmark of human intelligence, but challenging for reinforcement learning (RL) agents, is the ability to compositionally generalise, that is, to recompose familiar knowledge components in novel ways to solve new problems. For instance, when navigating in a city, one needs to know the location of the destination and how to operate a vehicle to get there, whether it be pedalling a bike or operating a car. In RL, these correspond to the reward function and transition function, respectively. To compositionally generalize, these two components need to be transferable independently of each other: multiple modes of transport can reach the same goal, and any given mode can be used to reach multiple destinations. Yet there are also instances where it can be helpful to learn and transfer entire structures, jointly representing goals and transitions, particularly whenever these recur in natural tasks (e.g., given a suggestion to get ice cream, one might prefer to bike, even in new towns). Prior theoretical work has explored how, in model-based RL, agents can learn and generalize task components (transition and reward functions). But a satisfactory account for how a single agent can simultaneously satisfy the two competing demands is still lacking. Here, we propose a hierarchical RL agent that learns and transfers individual task components as well as entire structures (particular compositions of components) by inferring both through a non-parametric Bayesian model of the task. It maintains a factorised representation of task components through a hierarchical Dirichlet process, but it also represents different possible covariances between these components through a standard Dirichlet process. We validate our approach on a variety of navigation tasks covering a wide range of statistical correlations between task components and show that it can also improve generalisation and transfer in more complex, hierarchical tasks with goal/subgoal structures. Finally, we end with a discussion of our work including how this clustering algorithm could conceivably be implemented by cortico-striatal gating circuits in the brain.

Implementing Online Reinforcement Learning with Clustering Neural Networks

Cluster-Based Social Reinforcement Learning

Implementing Online Reinforcement Learning with Temporal Neural Networks

Hierarchical clustering optimizes the tradeoff between compositionality and expressivity of task structures for flexible reinforcement learning

Clustered Reinforcement Learning

Neuromorphic Online Clustering and Classification

Large Scale Constrained Clustering With Reinforcement Learning

An Improved Reinforcement Learning Method Based on Unsupervised Learning

Cooperative Learning of Multi-Agent Systems Via Reinforcement Learning

Reinforcement Graph Clustering with Unknown Cluster Number

Deep Reinforcement Clustering

A Neuromorphic Architecture for Reinforcement Learning from Real-Valued Observations

Reinforcement Learning with Brain-Inspired Modulation can Improve Adaptation to Environmental Changes

Spiking neural networks with different reinforcement learning (RL) schemes in a multiagent setting

Meta Clustering of Neural Bandits

Inverse Reinforcement Learning from Non-Stationary Learning Agents

Adaptive Behavior Cloning Regularization for Stable Offline-to-Online Reinforcement Learning

Time-Efficient Reward Learning via Visually Assisted Cluster Ranking

A New Multi-Agent Reinforcement Learning Method Based on Evolving Dynamic Correlation Matrix

Clustering experience replay for the effective exploitation in reinforcement learning

An on-line algorithm for dynamic reinforcement learning and planning in reactive environments