Abstract:Imitation learning is a widely-used paradigm for decision making that learns from expert demonstrations. Existing imitation algorithms often require multiple interactions between the agent and the environment from which the demonstration is obtained. The acquisition of expert demonstrations in simulator usually requires specialized knowledge. In addition, real-world interactions are limited due to security or cost concerns. Therefore, the direct application of existing imitation learning algorithms in either real world or simulator is not an ideal strategy. In this paper, we propose a cross-domain Inverse Reinforcement Learning training paradigm that learns a reward function from hetero-domain expert’s demonstration, while the interaction with the environment that obtains demonstrations should be limited. In order to solve the distribution shift under such paradigm, we propose a transfer learning method called off-dynamics Inverse Reinforcement Learning. The intuition behind off-dynamics Inverse Reinforcement Learning is that the goal of reward function learning is not only to imitate experts, but also to promote action adaptation to the dynamic difference between two hetero-domain. Specifically, a widely-used Inverse Reinforcement Learning framework was adopted, and its discriminator for identifying agent-generated trajectories was modified with quantified dynamic differences. The training process of the discriminator yields the transferable reward function suitable for the target dynamics, which is guaranteed by our theoretical derivation. Off-dynamics Inverse Reinforcement Learning assigns higher rewards to demonstration trajectories that do not exploit discrepancies between the two domains. Our method demonstrates its effectiveness and scalability to high-dimensional tasks through extensive experiments on continuous control tasks. Our code is available on the project website: https://github.com/yachenkang/ODIRL.

Identifying Dynamic Regulation with Adversarial Surrogates

Off-Dynamics Inverse Reinforcement Learning

Analysis of the Identifying Regulation with Adversarial Surrogates Algorithm

[State Regulation for Complex Biological Networks Based on Dynamic Optimization Algorithms].

Adaptive Certainty-Equivalence Control With Regulation-Triggered Finite-Time Least-Squares Identification, Part I: Design

Identifying Latent Dynamic Components in Biological Systems

Simulating homeostatic, allostatic and goal-directed forms of interoceptive control using Active Inference

Regret-Optimal Defense Against Stealthy Adversaries: A System Level Approach

Data-Guided Regulator for Adaptive Nonlinear Control

Ambiguous Dynamic Treatment Regimes: A Reinforcement Learning Approach

Model-Free Incremental Adaptive Dynamic Programming Based Approximate Robust Optimal Regulation

Emergence of integrated behaviors through direct optimization for homeostasis

Steady-State Error Compensation in Reference Tracking and Disturbance Rejection Problems for Reinforcement Learning-Based Control

Data-Driven Adversarial Online Control for Unknown Linear Systems

Continuous Homeostatic Reinforcement Learning for Self-Regulated Autonomous Agents

Surprise! Using Physiological Stress for Allostatic Regulation Under the Active Inference Framework [Pre-Print]

Inverse Reinforcement Learning for Strategy Identification

A functional approach to homeostatic regulation

Adaptive Online Non-stochastic Control

Robust Adversarial Reinforcement Learning via Bounded Rationality Curricula

Data-Driven Robust Reinforcement Learning Control of Uncertain Nonlinear Systems: Towards a Fully-Automated, Insulin-Based Artificial Pancreas