Abstract:Although reinforcement learning methods offer a powerful framework for automatic skill acquisition, for practical learning-based control problems in domains such as robotics, imitation learning often provides a more convenient and accessible alternative. In particular, an interactive imitation learning method such as DAgger, which queries a near-optimal expert to intervene online to collect correction data for addressing the distributional shift challenges that afflict naïve behavioral cloning, can enjoy good performance both in theory and practice without requiring manually specified reward functions and other components of full reinforcement learning methods. In this paper, we explore how off-policy reinforcement learning can enable improved performance under assumptions that are similar but potentially even more practical than those of interactive imitation learning. Our proposed method uses reinforcement learning with user intervention signals themselves as rewards. This relaxes the assumption that intervening experts in interactive imitation learning should be near-optimal and enables the algorithm to learn behaviors that improve over the potential suboptimal human expert. We also provide a unified framework to analyze our RL method and DAgger; for which we present the asymptotic analysis of the suboptimal gap for both methods as well as the non-asymptotic sample complexity bound of our method. We then evaluate our method on challenging high-dimensional continuous control simulation benchmarks as well as real-world robotic vision-based manipulation tasks. The results show that it strongly outperforms DAgger-like approaches across the different tasks, especially when the intervening experts are suboptimal. Code and videos can be found on the project website: <a class="link-external link-https" href="https://rlif-page.github.io" rel="external noopener nofollow">this https URL</a>

PI-ELM: Reinforcement learning-based adaptable policy improvement for dynamical system

Robot Policy Improvement With Natural Evolution Strategies for Stable Nonlinear Dynamical System

Large Scale Pursuit-Evasion under Collision Avoidance Using Deep Reinforcement Learning.

Learning with Training Wheels: Speeding up Training with a Simple Controller for Deep Reinforcement Learning

Blending Imitation and Reinforcement Learning for Robust Policy Improvement

Safe Deep Policy Adaptation

RLIF: Interactive Imitation Learning as Reinforcement Learning

Mixed Reinforcement Learning for Efficient Policy Optimization in Stochastic Environments

Live in the Moment: Learning Dynamics Model Adapted to Evolving Policy

Improving Behavioural Cloning with Positive Unlabeled Learning

EvIL: Evolution Strategies for Generalisable Imitation Learning

Imitation-Enhanced Reinforcement Learning with Privileged Smooth Transition for Hexapod Locomotion

Accelerating Self-Imitation Learning from Demonstrations via Policy Constraints and Q-Ensemble

Error Bounds of Imitating Policies and Environments for Reinforcement Learning

RILe: Reinforced Imitation Learning

End-to-End Stable Imitation Learning via Autonomous Neural Dynamic Policies

Sample-Efficient Policy Learning Based on Completely Behavior Cloning.

Adaptive Evolutionary Reinforcement Learning with Policy Direction

OMPO: A Unified Framework for RL under Policy and Dynamics Shifts

Enabling Stateful Behaviors for Diffusion-based Policy Learning

Sim-to-Real Policy and Reward Transfer with Adaptive Forward Dynamics Model