Abstract:Although reinforcement learning methods offer a powerful framework for automatic skill acquisition, for practical learning-based control problems in domains such as robotics, imitation learning often provides a more convenient and accessible alternative. In particular, an interactive imitation learning method such as DAgger, which queries a near-optimal expert to intervene online to collect correction data for addressing the distributional shift challenges that afflict naïve behavioral cloning, can enjoy good performance both in theory and practice without requiring manually specified reward functions and other components of full reinforcement learning methods. In this paper, we explore how off-policy reinforcement learning can enable improved performance under assumptions that are similar but potentially even more practical than those of interactive imitation learning. Our proposed method uses reinforcement learning with user intervention signals themselves as rewards. This relaxes the assumption that intervening experts in interactive imitation learning should be near-optimal and enables the algorithm to learn behaviors that improve over the potential suboptimal human expert. We also provide a unified framework to analyze our RL method and DAgger; for which we present the asymptotic analysis of the suboptimal gap for both methods as well as the non-asymptotic sample complexity bound of our method. We then evaluate our method on challenging high-dimensional continuous control simulation benchmarks as well as real-world robotic vision-based manipulation tasks. The results show that it strongly outperforms DAgger-like approaches across the different tasks, especially when the intervening experts are suboptimal. Code and videos can be found on the project website: <a class="link-external link-https" href="https://rlif-page.github.io" rel="external noopener nofollow">this https URL</a>

Acquiring Diverse Skills using Curriculum Reinforcement Learning with Mixture of Experts

PDRL: Towards Deeper States and Further Behaviors in Unsupervised Skill Discovery by Progressive Diversity

Diffskill: Improving Reinforcement Learning Through Diffusion-Based Skill Denoiser for Robotic Manipulation

Non-local Policy Optimization via Diversity-regularized Collaborative Exploration

Robust Policy Learning via Offline Skill Diffusion

SkillS: Adaptive Skill Sequencing for Efficient Temporally-Extended Exploration

Accelerating Reinforcement Learning with Learned Skill Priors

Trying AGAIN instead of Trying Longer: Prior Learning for Automatic Curriculum Learning

Skill-Critic: Refining Learned Skills for Hierarchical Reinforcement Learning

RLIF: Interactive Imitation Learning as Reinforcement Learning

Unsupervised Reinforcement Learning for Multi-Task Autonomous Driving: Expanding Skills and Cultivating Curiosity

Diversity Progress for Goal Selection in Discriminability-Motivated RL

Curriculum-Based Imitation of Versatile Skills

Heterogeneous Skill Learning for Multi-agent Tasks

Skill matters: Dynamic skill learning for multi-agent cooperative reinforcement learning

Learning Multimodal Behaviors from Scratch with Diffusion Policy Gradient

Enhanced Generalization through Prioritization and Diversity in Self-Imitation Reinforcement Learning over Procedural Environments with Sparse Rewards

Density-based Curriculum for Multi-goal Reinforcement Learning with Sparse Rewards

Learning Diverse Policies with Soft Self-Generated Guidance

Exploration by Learning Diverse Skills through Successor State Measures

Generating Automatic Curricula via Self-Supervised Active Domain Randomization