Abstract:Imitation learning is a widely-used paradigm for decision making that learns from expert demonstrations. Existing imitation algorithms often require multiple interactions between the agent and the environment from which the demonstration is obtained. The acquisition of expert demonstrations in simulator usually requires specialized knowledge. In addition, real-world interactions are limited due to security or cost concerns. Therefore, the direct application of existing imitation learning algorithms in either real world or simulator is not an ideal strategy. In this paper, we propose a cross-domain Inverse Reinforcement Learning training paradigm that learns a reward function from hetero-domain expert’s demonstration, while the interaction with the environment that obtains demonstrations should be limited. In order to solve the distribution shift under such paradigm, we propose a transfer learning method called off-dynamics Inverse Reinforcement Learning. The intuition behind off-dynamics Inverse Reinforcement Learning is that the goal of reward function learning is not only to imitate experts, but also to promote action adaptation to the dynamic difference between two hetero-domain. Specifically, a widely-used Inverse Reinforcement Learning framework was adopted, and its discriminator for identifying agent-generated trajectories was modified with quantified dynamic differences. The training process of the discriminator yields the transferable reward function suitable for the target dynamics, which is guaranteed by our theoretical derivation. Off-dynamics Inverse Reinforcement Learning assigns higher rewards to demonstration trajectories that do not exploit discrepancies between the two domains. Our method demonstrates its effectiveness and scalability to high-dimensional tasks through extensive experiments on continuous control tasks. Our code is available on the project website: https://github.com/yachenkang/ODIRL.

Constraining the Size Growth of the Task Space with Socially Guided Intrinsic Motivation using Demonstrations

Bootstrapping Intrinsically Motivated Learning with Human Demonstrations

Socially Guided Intrinsic Motivation for Robot Learning of Motor Skills

Constrained-Space Optimization and Reinforcement Learning for Complex Tasks

Learning Generalizable 3D Manipulation With 10 Demonstrations

Screw Geometry Meets Bandits: Incremental Acquisition of Demonstrations to Generate Manipulation Plans

Skill Disentanglement for Imitation Learning from Suboptimal Demonstrations

Active Learning of Inverse Models with Intrinsically Motivated Goal Exploration in Robots

Off-Dynamics Inverse Reinforcement Learning

Comparing Skill Transfer Between Full Demonstrations and Segmented Sub-Tasks for Neural Dynamic Motion Primitives

Learning to Solve Tasks with Exploring Prior Behaviours

ZPD Teaching Strategies for Deep Reinforcement Learning from Demonstrations

Learning from Successful and Failed Demonstrations via Optimization

Show me the Way: Intrinsic Motivation from Demonstrations

Learning and generalization of task-parameterized skills through few human demonstrations

Modeling Long-horizon Tasks as Sequential Interaction Landscapes

Curriculum Design for Teaching via Demonstrations: Theory and Applications

Benchmark for Skill Learning from Demonstration: Impact of User Experience, Task Complexity, and Start Configuration on Performance

Learning Novel Skills from Language-Generated Demonstrations

Curriculum-Based Imitation of Versatile Skills

Unsupervised Learning of Goal Spaces for Intrinsically Motivated Goal Exploration