Abstract:In this paper, we study distributionally robust offline reinforcement learning (robust offline RL), which seeks to find an optimal policy purely from an offline dataset that can perform well in perturbed environments. In specific, we propose a generic algorithm framework called Doubly Pessimistic Model-based Policy Optimization ($P^2MPO$), which features a novel combination of a flexible model estimation subroutine and a doubly pessimistic policy optimization step. Notably, the double pessimism principle is crucial to overcome the distributional shifts incurred by (i) the mismatch between the behavior policy and the target policies; and (ii) the perturbation of the nominal model. Under certain accuracy conditions on the model estimation subroutine, we prove that $P^2MPO$ is sample-efficient with robust partial coverage data, which only requires the offline data to have good coverage of the distributions induced by the optimal robust policy and the perturbed models around the nominal model. By tailoring specific model estimation subroutines for concrete examples of RMDPs, including tabular RMDPs, factored RMDPs, kernel and neural RMDPs, we prove that $P^2MPO$ enjoys a $\tilde{\mathcal{O}}(n^{-1/2})$ convergence rate, where $n$ is the dataset size. We highlight that all these examples, except tabular RMDPs, are first identified and proven tractable by this work. Furthermore, we continue our study of robust offline RL in the robust Markov games (RMGs). By extending the double pessimism principle identified for single-agent RMDPs, we propose another algorithm framework that can efficiently find the robust Nash equilibria among players using only robust unilateral (partial) coverage data. To our best knowledge, this work proposes the first general learning principle -- double pessimism -- for robust offline RL and shows that it is provably efficient with general function approximation.

On Robustness of Double Linear Policy with Time-Varying Weights

On Robustness of Double Linear Trading with Transaction Costs

On Robust Optimal Linear Feedback Stock Trading

Robust Trading in a Generalized Lattice Market

Explicit Lipschitz Value Estimation Enhances Policy Robustness Against Perturbation

Robust stability for uncertain discrete singular systems with time-varying delays

Time Consistent Behavior Portfolio Policy for Dynamic Mean-Variance Formulation

Doubly Robust Interval Estimation for Optimal Policy Evaluation in Online Learning

First-order Policy Optimization for Robust Policy Evaluation

Robust Policy Optimization in Continuous-time Mixed $\mathcal{H}_2/\mathcal{H}_\infty$ Stochastic Control

Robust Equilibrium Control-Measure Policy for a DC Pension Plan with State-Dependent Risk Aversion under Mean-Variance Criterion

Efficient and Sharp Off-Policy Evaluation in Robust Markov Decision Processes

Robust Time-inconsistent Linear-Quadratic Stochastic Controls: A Stochastic Differential Game Approach

Soft Policy Optimization Using Dual-Track Advantage Estimator.

Efficient Policy Iteration for Robust Markov Decision Processes via Regularization

Blackwell optimality and policy stability for long-run risk sensitive stochastic control

Double Pessimism is Provably Efficient for Distributionally Robust Offline Reinforcement Learning: Generic Algorithm and Robust Partial Coverage

Designing Tractable Piecewise Affine Policies for Multi-Stage Adjustable Robust Optimization

On Robust Stability for Uncertain Time-Delay Systems: A Polyhedral Lyapunov–Krasovskii Approach

H_∞ Model-free Reinforcement Learning with Robust Stability Guarantee

A Single-Loop Robust Policy Gradient Method for Robust Markov Decision Processes