Abstract:Offline inverse reinforcement learning (Offline IRL) aims to recover the structure of rewards and environment dynamics that underlie observed actions in a fixed, finite set of demonstrations from an expert agent. Accurate models of expertise in executing a task has applications in safety-sensitive applications such as clinical decision making and autonomous driving. However, the structure of an expert's preferences implicit in observed actions is closely linked to the expert's model of the environment dynamics (i.e. the ``world'' model). Thus, inaccurate models of the world obtained from finite data with limited coverage could compound inaccuracy in estimated rewards. To address this issue, we propose a bi-level optimization formulation of the estimation task wherein the upper level is likelihood maximization based upon a conservative model of the expert's policy (lower level). The policy model is conservative in that it maximizes reward subject to a penalty that is increasing in the uncertainty of the estimated model of the world. We propose a new algorithmic framework to solve the bi-level optimization problem formulation and provide statistical and computational guarantees of performance for the associated optimal reward estimator. Finally, we demonstrate that the proposed algorithm outperforms the state-of-the-art offline IRL and imitation learning benchmarks by a large margin, over the continuous control tasks in MuJoCo and different datasets in the D4RL benchmark.

What problem does this paper attempt to address?

### Problems the Paper Aims to Solve This paper aims to address the issue of distribution shift in Offline Inverse Reinforcement Learning (Offline IRL) and to recover high-quality reward functions from a limited set of demonstration data. Specifically, the paper focuses on the following points: 1. **Challenges of Offline Inverse Reinforcement Learning**: Traditional inverse reinforcement learning methods require a large amount of online interaction to recover reward functions and imitate expert behavior. However, in many practical applications (such as clinical decision-making and autonomous driving), such online trial and error is not feasible due to safety concerns. Therefore, offline inverse reinforcement learning avoids the need for online environment interaction by utilizing only a fixed set of demonstration data. 2. **Distribution Shift Problem**: In offline inverse reinforcement learning, the reward functions and policies recovered from a limited dataset have poor generalization ability in new states, leading to suboptimal performance. This is because the dataset has limited coverage of the true environment dynamics, resulting in accumulated estimation errors. 3. **Conservative Policy Training**: Drawing from advances in the field of offline reinforcement learning, the paper proposes a method that combines conservative policy training to avoid the problem of value function overestimation caused by distribution shift. ### Research Contributions 1. **Maximum Likelihood Framework**: The paper proposes a two-stage algorithm framework based on Maximum Likelihood Estimation (MLE). The first stage estimates the environment dynamics model (world model), and the second stage recovers the reward function in a conservative Markov decision process. 2. **Efficient Algorithm**: An efficient algorithm is proposed to solve the offline inverse reinforcement learning problem, avoiding the need to repeatedly solve the policy optimization problem for each reward estimation. The algorithm alternates between one reward update and one conservative policy improvement. 3. **Statistical and Computational Guarantees**: The paper provides theoretical guarantees in terms of statistics and computation, proving that the algorithm converges to an approximate stable point within a finite time and can find the global optimal solution when the reward function is linearly parameterized. Through the above methods, the paper demonstrates that the proposed algorithm significantly outperforms existing offline inverse reinforcement learning and imitation learning methods in MuJoCo robotic control tasks and D4RL benchmarks.

When Demonstrations Meet Generative World Models: A Maximum Likelihood Framework for Offline Inverse Reinforcement Learning

Off-Dynamics Inverse Reinforcement Learning

Offline Inverse RL: New Solution Concepts and Provably Efficient Algorithms

Maximum-Likelihood Inverse Reinforcement Learning with Finite-Time Guarantees

A Bayesian Approach to Robust Inverse Reinforcement Learning

Bridging Imitation and Online Reinforcement Learning: An Optimistic Tale

Finetuning Offline World Models in the Real World

Offline Imitation Learning with Model-based Reverse Augmentation

Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism

CLARE: Conservative Model-Based Reward Learning for Offline Inverse Reinforcement Learning

Offline Reinforcement Learning with Reverse Model-based Imagination

Making Offline RL Online: Collaborative World Models for Offline Visual Reinforcement Learning

A Framework and Method for Online Inverse Reinforcement Learning

A Unified Linear Programming Framework for Offline Reward Learning from Human Demonstrations and Feedback

Towards Data-Driven Offline Simulations for Online Reinforcement Learning

Offline Reinforcement Learning via High-Fidelity Generative Behavior Modeling

A Study of Continuous Maximum Entropy Deep Inverse Reinforcement Learning

Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning

Optimality Inductive Biases and Agnostic Guidelines for Offline Reinforcement Learning

Maximum Likelihood Constraint Inference for Inverse Reinforcement Learning

Survival Instinct in Offline Reinforcement Learning