Abstract:Imitation learning is a widely-used paradigm for decision making that learns from expert demonstrations. Existing imitation algorithms often require multiple interactions between the agent and the environment from which the demonstration is obtained. The acquisition of expert demonstrations in simulator usually requires specialized knowledge. In addition, real-world interactions are limited due to security or cost concerns. Therefore, the direct application of existing imitation learning algorithms in either real world or simulator is not an ideal strategy. In this paper, we propose a cross-domain Inverse Reinforcement Learning training paradigm that learns a reward function from hetero-domain expert’s demonstration, while the interaction with the environment that obtains demonstrations should be limited. In order to solve the distribution shift under such paradigm, we propose a transfer learning method called off-dynamics Inverse Reinforcement Learning. The intuition behind off-dynamics Inverse Reinforcement Learning is that the goal of reward function learning is not only to imitate experts, but also to promote action adaptation to the dynamic difference between two hetero-domain. Specifically, a widely-used Inverse Reinforcement Learning framework was adopted, and its discriminator for identifying agent-generated trajectories was modified with quantified dynamic differences. The training process of the discriminator yields the transferable reward function suitable for the target dynamics, which is guaranteed by our theoretical derivation. Off-dynamics Inverse Reinforcement Learning assigns higher rewards to demonstration trajectories that do not exploit discrepancies between the two domains. Our method demonstrates its effectiveness and scalability to high-dimensional tasks through extensive experiments on continuous control tasks. Our code is available on the project website: https://github.com/yachenkang/ODIRL.

Skill based transfer learning with domain adaptation for continuous reinforcement learning domains

Transfer Learning and Deep Domain Adaptation

Off-Dynamics Inverse Reinforcement Learning

Graph based skill acquisition and transfer Learning for continuous reinforcement learning domains

Transferring knowledge from human-demonstration trajectories to reinforcement learning

Cross-domain adaptive transfer reinforcement learning based on state-action correspondence.

Domain Adaptation In Reinforcement Learning Via Latent Unified State Representation

Deep Reinforcement Learning for Autonomous Driving by Transferring Visual Features.

Shaping in Reinforcement Learning Via Knowledge Transferred from Human-Demonstrations

Domain Adaptive State Representation Alignment for Reinforcement Learning

Domain Adaptation for Reinforcement Learning on the Atari

Task and Domain Adaptive Reinforcement Learning for Robot Control

Knowledge Transfer for Cross-Domain Reinforcement Learning: A Systematic Review

Domain Adaptation Through Task Distillation

Transferring Domain Knowledge with an Adviser in Continuous Tasks

Transfer Learning in Deep Reinforcement Learning: A Survey

SCRL: Self-supervised Continual Reinforcement Learning for Domain Adaptation

Heuristically Accelerated Reinforcement Learning by Means of Case-Based Reasoning and Transfer Learning

Cross-Domain Communications Between Agents Via Adversarial-Based Domain Adaptation in Reinforcement Learning

Transfer in Deep Reinforcement Learning Using Successor Features and Generalised Policy Improvement

Adaptive Meta-Domain Transfer Learning (AMDTL): A Novel Approach for Knowledge Transfer in AI