Abstract:A hallmark of human intelligence, but challenging for reinforcement learning (RL) agents, is the ability to compositionally generalise, that is, to recompose familiar knowledge components in novel ways to solve new problems. For instance, when navigating in a city, one needs to know the location of the destination and how to operate a vehicle to get there, whether it be pedalling a bike or operating a car. In RL, these correspond to the reward function and transition function, respectively. To compositionally generalize, these two components need to be transferable independently of each other: multiple modes of transport can reach the same goal, and any given mode can be used to reach multiple destinations. Yet there are also instances where it can be helpful to learn and transfer entire structures, jointly representing goals and transitions, particularly whenever these recur in natural tasks (e.g., given a suggestion to get ice cream, one might prefer to bike, even in new towns). Prior theoretical work has explored how, in model-based RL, agents can learn and generalize task components (transition and reward functions). But a satisfactory account for how a single agent can simultaneously satisfy the two competing demands is still lacking. Here, we propose a hierarchical RL agent that learns and transfers individual task components as well as entire structures (particular compositions of components) by inferring both through a non-parametric Bayesian model of the task. It maintains a factorised representation of task components through a hierarchical Dirichlet process, but it also represents different possible covariances between these components through a standard Dirichlet process. We validate our approach on a variety of navigation tasks covering a wide range of statistical correlations between task components and show that it can also improve generalisation and transfer in more complex, hierarchical tasks with goal/subgoal structures. Finally, we end with a discussion of our work including how this clustering algorithm could conceivably be implemented by cortico-striatal gating circuits in the brain.

Neural Constraint Satisfaction: Hierarchical Abstraction for Combinatorial Generalization in Object Rearrangement

Adversarial Object Rearrangement in Constrained Environments with Heterogeneous Graph Neural Networks

Structural Concept Learning via Graph Attention for Multi-Level Rearrangement Planning

Learning Object Relation Graph and Goal-Aware Policy for Visual Room Rearrangement

ConSOR: A Context-Aware Semantic Object Rearrangement Framework for Partially Arranged Scenes

"Set It Up!": Functional Object Arrangement with Compositional Generative Models

SG-Bot: Object Rearrangement via Coarse-to-Fine Robotic Imagination on Scene Graphs

Hierarchical clustering optimizes the tradeoff between compositionality and expressivity of task structures for flexible reinforcement learning

Stimulating Imagination: Towards General-purpose Object Rearrangement

Hierarchical Policy for Non-prehensile Multi-object Rearrangement with Deep Reinforcement Learning and Monte Carlo Tree Search

Visual Room Rearrangement

CabiNet: Scaling Neural Collision Detection for Object Rearrangement with Procedural Scene Generation

Entity-Centric Reinforcement Learning for Object Manipulation from Pixels

3D reconstruction based on hierarchical reinforcement learning with transferability

MANER: Multi-Agent Neural Rearrangement Planning of Objects in Cluttered Environments

Object Rearrangement Using Learned Implicit Collision Functions

Monte-Carlo Tree Search for Efficient Visually Guided Rearrangement Planning

Robo-ABC: Affordance Generalization Beyond Categories via Semantic Correspondence for Robot Manipulation

Latent Space Planning for Multiobject Manipulation With Environment-Aware Relational Classifiers

SplatR : Experience Goal Visual Rearrangement with 3D Gaussian Splatting and Dense Feature Matching

Latent Space Planning for Multi-Object Manipulation with Environment-Aware Relational Classifiers