Abstract:The performance of imitation learning is typically upper-bounded by the performance of the demonstrator. While recent empirical results demonstrate that ranked demonstrations allow for better-than-demonstrator performance, preferences over demonstrations may be difficult to obtain, and little is known theoretically about when such methods can be expected to successfully extrapolate beyond the performance of the demonstrator. To address these issues, we first contribute a sufficient condition for better-than-demonstrator imitation learning and provide theoretical results showing why preferences over demonstrations can better reduce reward function ambiguity when performing inverse reinforcement learning. Building on this theory, we introduce Disturbance-based Reward Extrapolation (D-REX), a ranking-based imitation learning method that injects noise into a policy learned through behavioral cloning to automatically generate ranked demonstrations. These ranked demonstrations are used to efficiently learn a reward function that can then be optimized using reinforcement learning. We empirically validate our approach on simulated robot and Atari imitation learning benchmarks and show that D-REX outperforms standard imitation learning approaches and can significantly surpass the performance of the demonstrator. D-REX is the first imitation learning approach to achieve significant extrapolation beyond the demonstrator's performance without additional side-information or supervision, such as rewards or human preferences. By generating rankings automatically, we show that preference-based inverse reinforcement learning can be applied in traditional imitation learning settings where only unlabeled demonstrations are available.

Imitation Learning from Suboptimal Demonstrations Via Discriminator Weighting

Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrations.

Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrations

Imitation Learning from Suboptimal Demonstrations via Meta-Learning An Action Ranker

Imitation Learning from Imperfection: Theoretical Justifications and Algorithms

Learning to Discern: Imitating Heterogeneous Human Demonstrations with Preference and Representation Learning

Imitation Learning from Purified Demonstrations

Imitation Learning to Outperform Demonstrators by Directly Extrapolating Demonstrations

Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning

Better-than-Demonstrator Imitation Learning via Automatically-Ranked Demonstrations

Learning to Weight Imperfect Demonstrations.

Extrinsicaly Rewarded Soft Q Imitation Learning with Discriminator

Unlabeled Imperfect Demonstrations in Adversarial Imitation Learning

Learning Feasibility to Imitate Demonstrators with Different Dynamics

Skill Disentanglement for Imitation Learning from Suboptimal Demonstrations

Efficient Off-policy Adversarial Imitation Learning with Imperfect Demonstrations

How to Leverage Diverse Demonstrations in Offline Imitation Learning

Learning from Suboptimal Demonstration via Self-Supervised Reward Regression

Robust Adversarial Imitation Learning Via Adaptively-Selected Demonstrations

Learning Noise-Induced Reward Functions for Surpassing Demonstrations in Imitation Learning.

Towards Improving Learning from Demonstration Algorithms via MCMC Methods