Abstract:In offline Imitation Learning (IL), one of the main challenges is the \textit{covariate shift} between the expert observations and the actual distribution encountered by the agent, because it is difficult to determine what action an agent should take when outside the state distribution of the expert demonstrations. Recently, the model-free solutions introduce the supplementary data and identify the latent expert-similar samples to augment the reliable samples during learning. Model-based solutions build forward dynamic models with conservatism quantification and then generate additional trajectories in the neighborhood of expert demonstrations. However, without reward supervision, these methods are often over-conservative in the out-of-expert-support regions, because only in states close to expert-observed states can there be a preferred action enabling policy optimization. To encourage more exploration on expert-unobserved states, we propose a novel model-based framework, called offline Imitation Learning with Self-paced Reverse Augmentation (SRA). Specifically, we build a reverse dynamic model from the offline demonstrations, which can efficiently generate trajectories leading to the expert-observed states in a self-paced style. Then, we use the subsequent reinforcement learning method to learn from the augmented trajectories and transit from expert-unobserved states to expert-observed states. This framework not only explores the expert-unobserved states but also guides maximizing long-term returns on these states, ultimately enabling generalization beyond the expert data. Empirical results show that our proposal could effectively mitigate the covariate shift and achieve the state-of-the-art performance on the offline imitation learning benchmarks. Project website: \url{<a class="link-external link-https" href="https://www.lamda.nju.edu.cn/shaojj/KDD24_SRA/" rel="external noopener nofollow">this https URL</a>}.

Run Away From your Teacher: Understanding BYOL by a Novel Self-Supervised Approach

Bootstrap your own latent: A new approach to self-supervised Learning

Hyperspherically Regularized Networks for Self-Supervision

The Edge of Orthogonality: A Simple View of What Makes BYOL Tick

Understanding self-supervised Learning Dynamics without Contrastive Pairs

Learning Where to Learn in Cross-View Self-Supervised Learning

ROBYOL: Random-Occlusion-Based BYOL for Hyperspectral Image Classification

Adaptive Similarity Bootstrapping for Self-Distillation based Representation Learning

Self-Training with Differentiable Teacher

BYOL for Audio: Self-Supervised Learning for General-Purpose Audio Representation

Towards a Unified Theoretical Understanding of Non-contrastive Learning via Rank Differential Mechanism

Consensus Clustering With Unsupervised Representation Learning

LTRL: Boosting Long-tail Recognition via Reflective Learning

RA-YOLOX: Re-parameterization align decoupled head and novel label assignment scheme based on YOLOX

RSA: Reducing Semantic Shift from Aggressive Augmentations for Self-supervised Learning

Robust and Discriminative Self-Taught Learning.

Self-Damaging Contrastive Learning

Offline Imitation Learning with Model-based Reverse Augmentation

BYOL for Audio: Exploring Pre-Trained General-Purpose Audio Representations

Solving Inefficiency of Self-supervised Representation Learning

Occam's Razor for Self Supervised Learning: What is Sufficient to Learn Good Representations?