Abstract:Imitation learning is a widely-used paradigm for decision making that learns from expert demonstrations. Existing imitation algorithms often require multiple interactions between the agent and the environment from which the demonstration is obtained. The acquisition of expert demonstrations in simulator usually requires specialized knowledge. In addition, real-world interactions are limited due to security or cost concerns. Therefore, the direct application of existing imitation learning algorithms in either real world or simulator is not an ideal strategy. In this paper, we propose a cross-domain Inverse Reinforcement Learning training paradigm that learns a reward function from hetero-domain expert’s demonstration, while the interaction with the environment that obtains demonstrations should be limited. In order to solve the distribution shift under such paradigm, we propose a transfer learning method called off-dynamics Inverse Reinforcement Learning. The intuition behind off-dynamics Inverse Reinforcement Learning is that the goal of reward function learning is not only to imitate experts, but also to promote action adaptation to the dynamic difference between two hetero-domain. Specifically, a widely-used Inverse Reinforcement Learning framework was adopted, and its discriminator for identifying agent-generated trajectories was modified with quantified dynamic differences. The training process of the discriminator yields the transferable reward function suitable for the target dynamics, which is guaranteed by our theoretical derivation. Off-dynamics Inverse Reinforcement Learning assigns higher rewards to demonstration trajectories that do not exploit discrepancies between the two domains. Our method demonstrates its effectiveness and scalability to high-dimensional tasks through extensive experiments on continuous control tasks. Our code is available on the project website: https://github.com/yachenkang/ODIRL.

Nonuniqueness and Convergence to Equivalent Solutions in Observer-based Inverse Reinforcement Learning

Convergence Analysis of an Incremental Approach to Online Inverse Reinforcement Learning

Off-Dynamics Inverse Reinforcement Learning

Off-Dynamics Inverse Reinforcement Learning from Hetero-Domain

Online Observer-Based Inverse Reinforcement Learning

A Framework and Method for Online Inverse Reinforcement Learning

Event-based Integral Reinforcement Learning Algorithm for Non-zero-sum Games of Partially Unknown Nonlinear Systems

Offline Inverse RL: New Solution Concepts and Provably Efficient Algorithms

Integral Reinforcement Learning-Based Online Adaptive Event-Triggered Control for Non-Zero-sum Games of Partially Unknown Nonlinear Systems.

Towards Theoretical Understanding of Inverse Reinforcement Learning

Hybrid Inverse Reinforcement Learning

When Demonstrations Meet Generative World Models: A Maximum Likelihood Framework for Offline Inverse Reinforcement Learning

Maximum-Likelihood Inverse Reinforcement Learning with Finite-Time Guarantees

Online Stackelberg Optimization via Nonlinear Control

How does Inverse RL Scale to Large State Spaces? A Provably Efficient Approach

Off-policy Synchronous Iteration IRL Method for Multi-Player Zero-Sum Games with Input Constraints

Integral Reinforcement Learning Off-Policy Method for Solving Nonlinear Multi-Player Nonzero-Sum Games with Saturated Actuator.

A Survey of Inverse Reinforcement Learning Techniques.

Reinforcement Learning Based Solution To Two-Player Zero-Sum Game Using Differentiator

Data-Driven Inverse Reinforcement Learning for Expert-Learner Zero-Sum Games

Online adaptive data-driven control for unknown nonlinear systems with constrained-input