Abstract: Inverse reinforcement learning (IRL) aims to recover the reward function and the associated optimal policy that best fits observed sequences of states and actions implemented by an expert. Many algorithms for IRL have an inherently nested structure: the inner loop finds the optimal policy given parametrized rewards while the outer loop updates the estimates towards optimizing a measure of fit. For high dimensional environments such nested-loop structure entails a significant computational burden. To reduce the computational burden of a nested loop, novel methods such as SQIL [1] and IQ-Learn [2] emphasize policy estimation at the expense of reward estimation accuracy. However, without accurate estimated rewards, it is not possible to do counterfactual analysis such as predicting the optimal policy under different environment dynamics and/or learning new tasks. In this paper we develop a novel single-loop algorithm for IRL that does not compromise reward estimation accuracy. In the proposed algorithm, each policy improvement step is followed by a stochastic gradient step for likelihood maximization. We show that the proposed algorithm provably converges to a stationary solution with a finite-time guarantee. If the reward is parameterized linearly, we show the identified solution corresponds to the solution of the maximum entropy IRL problem. Finally, by using robotics control problems in MuJoCo and their transfer settings, we show that the proposed algorithm achieves superior performance compared with other IRL and imitation learning benchmarks.

Inverse reinforcement learning by expert imitation for the stochastic linear-quadratic optimal control problem

Inverse Reinforcement Q-Learning Through Expert Imitation for Discrete-Time Systems

Model-Free Inverse H-Infinity Control for Imitation Learning

Optimal Control for Constrained Discrete-Time Nonlinear Systems Based on Safe Reinforcement Learning.

Convergence Analysis of an Incremental Approach to Online Inverse Reinforcement Learning

Stable Inverse Reinforcement Learning: Policies from Control Lyapunov Landscapes

Data-Driven Inverse Reinforcement Learning for Expert-Learner Zero-Sum Games

Hybrid Inverse Reinforcement Learning

Inverse Reinforcement Learning for Discrete-Time Linear Quadratic Systems

Inverse Reinforcement Learning with Sub-optimal Experts

Inverse Reinforcement Learning for Trajectory Imitation Using Static Output Feedback Control

Robust policy iteration for continuous-time stochastic $H_\infty$ control problem with unknown dynamics

Reinforcement Learning-Based Control for Nonlinear Discrete-Time Systems with Unknown Control Directions and Control Constraints

Online Observer-Based Inverse Reinforcement Learning

Inverse Reinforcement Learning with Explicit Policy Estimates

Inverse Reinforcement Learning in Tracking Control Based on Inverse Optimal Control

Non-Adversarial Inverse Reinforcement Learning via Successor Feature Matching

Synchronous Optimal Control Method for Nonlinear Systems with Saturating Actuators and Unknown Dynamics Using Off-Policy Integral Reinforcement Learning

Maximum-Likelihood Inverse Reinforcement Learning with Finite-Time Guarantees

Generalized Policy Iteration-based Reinforcement Learning Algorithm for Optimal Control of Unknown Discrete-time Systems

Curricular Subgoals for Inverse Reinforcement Learning