Abstract:Imitation learning is a widely-used paradigm for decision making that learns from expert demonstrations. Existing imitation algorithms often require multiple interactions between the agent and the environment from which the demonstration is obtained. The acquisition of expert demonstrations in simulator usually requires specialized knowledge. In addition, real-world interactions are limited due to security or cost concerns. Therefore, the direct application of existing imitation learning algorithms in either real world or simulator is not an ideal strategy. In this paper, we propose a cross-domain Inverse Reinforcement Learning training paradigm that learns a reward function from hetero-domain expert’s demonstration, while the interaction with the environment that obtains demonstrations should be limited. In order to solve the distribution shift under such paradigm, we propose a transfer learning method called off-dynamics Inverse Reinforcement Learning. The intuition behind off-dynamics Inverse Reinforcement Learning is that the goal of reward function learning is not only to imitate experts, but also to promote action adaptation to the dynamic difference between two hetero-domain. Specifically, a widely-used Inverse Reinforcement Learning framework was adopted, and its discriminator for identifying agent-generated trajectories was modified with quantified dynamic differences. The training process of the discriminator yields the transferable reward function suitable for the target dynamics, which is guaranteed by our theoretical derivation. Off-dynamics Inverse Reinforcement Learning assigns higher rewards to demonstration trajectories that do not exploit discrepancies between the two domains. Our method demonstrates its effectiveness and scalability to high-dimensional tasks through extensive experiments on continuous control tasks. Our code is available on the project website: https://github.com/yachenkang/ODIRL.

Domain transfer via cross-domain analogy

Off-Dynamics Inverse Reinforcement Learning

A New Domain Adaptation Method Based on Rules Discovered from Cross-Domain Features

Domain Adaption for Knowledge Tracing

Domain Adaptation via Transfer Component Analysis

Implicit Analogies in Learning: Supporting Transfer by Warming Up

Analogical Learning in Tactical Decision Games

Dictionary Transfer for Image Denoising Via Domain Adaptation

Domain adaptive knowledge tracing

Domain Adaptation Through Task Distillation

Visualizing Transferred Knowledge: An Interpretive Model of Unsupervised Domain Adaptation

Few-Shot Text and Image Classification via Analogical Transfer Learning

Non-linear Domain Adaptation in Transfer Evolutionary Optimization

Tackling Morphological Analogies Using Deep Learning -- Extended Version

Visual Analogies between Atari Games for Studying Transfer Learning in RL

Adaptively-Accumulated Knowledge Transfer for Partial Domain Adaptation

Cross-Domain Kernel Induction for Transfer Learning

Theory-based Causal Transfer: Integrating Instance-level Induction and Abstract-level Structure Learning

Measuring the Feasibility of Analogical Transfer using Complexity

Analogical reasoning during hypothesis generation: the effects of object and domain similarities on access and transfer

Unsupervised Transductive Domain Adaptation