Abstract:We study ObjectGoal Navigation - where a virtual robot situated in a new environment is asked to navigate to an object. Prior work [1] has shown that imitation learning (IL) using behavior cloning (BC) on a dataset of human demonstrations achieves promising results. However, this has limitations - 1) BC policies generalize poorly to new states, since the training mimics actions not their consequences, and 2) collecting demonstrations is expensive. On the other hand, reinforcement learning (RL) is trivially scalable, but requires careful reward engineering to achieve desirable behavior. We present PIRLNav, a two-stage learning scheme for BC pretraining on human demonstrations followed by RL-finetuning. This leads to a policy that achieves a success rate of 65.0% on OBJECTNAV (+5.0% absolute over previous state-of-the-art). Using this BC→RL training recipe, we present a rigorous empirical analysis of design choices. First, we investigate whether human demonstrations can be replaced with ‘free’ (automatically generated) sources of demonstrations, e.g. shortest paths (SP) or task-agnostic frontier exploration (FE) trajectories. We find that BC→RL on human demonstrations outperforms BC→RL on SP and FE trajectories, even when controlled for the same BC-pretraining success on TRAIN, and even on a subset of VAL episodes where BC-pretraining success favors the SP or FE policies. Next, we study how RL-finetuning performance scales with the size of the BC pretraining dataset. We find that as we increase the size of the BC-pretraining dataset and get to high BC accuracies, the improvements from RL-finetuning are smaller, and that 90% of the performance of our best BC→RL policy can be achieved with less than half the number of BC demonstrations. Finally, we analyze failure modes of our OBJECTNAV policies, and present guidelines for further improving them. Project page: ram81.github.io/projects/pirlnav.

PLANRL: A Motion Planning and Imitation Learning Framework to Bootstrap Reinforcement Learning

RL and ANN Based Modular Path Planning Controller for Resource-Constrained Robots in the Indoor Complex Dynamic Environment

Using Implicit Behavior Cloning and Dynamic Movement Primitive to Facilitate Reinforcement Learning for Robot Motion Planning

Learning with Training Wheels: Speeding up Training with a Simple Controller for Deep Reinforcement Learning

A RDA-Based Deep Reinforcement Learning Approach for Autonomous Motion Planning of UAV in Dynamic Unknown Environments

LIRL: Latent Imagination-Based Reinforcement Learning for Efficient Coverage Path Planning

A Reinforcement Learning-Boosted Motion Planning Framework: Comprehensive Generalization Performance in Autonomous Driving

SigmaRL: A Sample-Efficient and Generalizable Multi-Agent Reinforcement Learning Framework for Motion Planning

Learning to Bridge the Gap: Efficient Novelty Recovery with Planning and Reinforcement Learning

Hierarchies of Planning and Reinforcement Learning for Robot Navigation

Efficient Learning of High Level Plans from Play

RL-RRT: Kinodynamic Motion Planning via Learning Reachability Estimators from RL Policies

SPIRE: Synergistic Planning, Imitation, and Reinforcement Learning for Long-Horizon Manipulation

Reinforced Imitative Trajectory Planning for Urban Automated Driving

PIRLNav: Pretraining with Imitation and RL Finetuning for OBJECTNAV

Plan-Seq-Learn: Language Model Guided RL for Solving Long Horizon Robotics Tasks

On Reward Shaping for Mobile Robot Navigation: A Reinforcement Learning and SLAM Based Approach

Imitation Bootstrapped Reinforcement Learning

iNAV-drlSLAM: An Improved Indoor Self-Driving Framework for Mobile Robots using Deep Reinforcement Learning integrated with SLAM

Learning whom to trust in navigation: dynamically switching between classical and neural planning