Abstract:The real world is unpredictable. Therefore, to solve long-horizon decision-making problems with autonomous robots, we must construct agents that are capable of adapting to changes in the environment during deployment. Model-based planning approaches can enable robots to solve complex, long-horizon tasks in a variety of environments. However, such approaches tend to be brittle when deployed into an environment featuring a novel situation that their underlying model does not account for. In this work, we propose to learn a ``bridge policy'' via Reinforcement Learning (RL) to adapt to such novelties. We introduce a simple formulation for such learning, where the RL problem is constructed with a special ``CallPlanner'' action that terminates the bridge policy and hands control of the agent back to the planner. This allows the RL policy to learn the set of states in which querying the planner and following the returned plan will achieve the goal. We show that this formulation enables the agent to rapidly learn by leveraging the planner's knowledge to avoid challenging long-horizon exploration caused by sparse reward. In experiments across three different simulated domains of varying complexity, we demonstrate that our approach is able to learn policies that adapt to novelty more efficiently than several baselines, including a pure RL baseline. We also demonstrate that the learned bridge policy is generalizable in that it can be combined with the planner to enable the agent to solve more complex tasks with multiple instances of the encountered novelty.

Planning for Novelty: Width-Based Algorithms for Common Problems in Control, Planning and Reinforcement Learning

Count-based Novelty Exploration in Classical Planning

Efficient Novelty Search Through Deep Reinforcement Learning

Learning to Bridge the Gap: Efficient Novelty Recovery with Planning and Reinforcement Learning

Novelty and Lifted Helpful Actions in Generalized Planning

Best-First Width Search for Multi Agent Privacy-preserving Planning

Novelty Accommodating Multi-Agent Planning in High Fidelity Simulated Open World

Learning to Operate in Open Worlds by Adapting Planning Models

Section Patterns: Efficiently Solving Narrow Passage Problems in Multilevel Motion Planning

Path Planning in Dynamic Environments with Adaptive Dimensionality

Can Vehicle Motion Planning Generalize to Realistic Long-tail Scenarios?

Proximity-Based Non-uniform Abstractions for Approximate Planning

Expanding horizons in reinforcement learning for curious exploration and creative planning

Motion Planning: A Journey of Robots, Molecules, Digital Actors, and Other Artifacts

NovGrid: A Flexible Grid World for Evaluating Agent Response to Novelty

Novelty Detection in Reinforcement Learning with World Models

Integrating Planning, Execution and Monitoring in the presence of Open World Novelties: Case Study of an Open World Monopoly Solver

NovelGym: A Flexible Ecosystem for Hybrid Planning and Learning Agents Designed for Open Worlds

A Contextual Bandit Approach for Learning to Plan in Environments with Probabilistic Goal Configurations

Generative Planning with Fast Collision Checks for High Speed Navigation

Learning to Plan in High Dimensions via Neural Exploration-Exploitation Trees