Abstract:We study evaluating a policy under best- and worst-case perturbations to a Markov decision process (MDP), given transition observations from the original MDP, whether under the same or different policy. This is an important problem when there is the possibility of a shift between historical and future environments, due to e.g. unmeasured confounding, distributional shift, or an adversarial environment. We propose a perturbation model that can modify transition kernel densities up to a given multiplicative factor or its reciprocal, which extends the classic marginal sensitivity model (MSM) for single time step decision making to infinite-horizon RL. We characterize the sharp bounds on policy value under this model, that is, the tightest possible bounds given by the transition observations from the original MDP, and we study the estimation of these bounds from such transition observations. We develop an estimator with several appealing guarantees: it is semiparametrically efficient, and remains so even when certain necessary nuisance functions such as worst-case Q-functions are estimated at slow nonparametric rates. It is also asymptotically normal, enabling easy statistical inference using Wald confidence intervals. In addition, when certain nuisances are estimated inconsistently we still estimate a valid, albeit possibly not sharp bounds on the policy value. We validate these properties in numeric simulations. The combination of accounting for environment shifts from train to test (robustness), being insensitive to nuisance-function estimation (orthogonality), and accounting for having only finite samples to learn from (inference) together leads to credible and reliable policy evaluation.

Efficient Evaluation of Natural Stochastic Policies in Offline Reinforcement Learning

Beyond Reward: Offline Preference-guided Policy Optimization

Efficient Policy Evaluation with Offline Data Informed Behavior Policy Design

OPERA: Automatic Offline Policy Evaluation with Re-weighted Aggregates of Multiple Estimators

Offline Policy Evaluation for Reinforcement Learning with Adaptively Collected Data

Efficient Multi-Policy Evaluation for Reinforcement Learning

More Efficient Off-Policy Evaluation through Regularized Targeted Learning

Robust On-Policy Sampling for Data-Efficient Policy Evaluation in Reinforcement Learning

Efficient Offline Reinforcement Learning: The Critic is Critical

Offline Policy Comparison under Limited Historical Agent-Environment Interactions

Stable and Efficient Policy Evaluation.

Efficient Policy Evaluation with Safety Constraint for Reinforcement Learning

$K$-Nearest-Neighbor Resampling for Off-Policy Evaluation in Stochastic Control

Offline Policy Evaluation and Optimization under Confounding

Reliable Off-policy Evaluation for Reinforcement Learning

Off-policy Evaluation in Infinite-Horizon Reinforcement Learning with Latent Confounders

Matrix Estimation for Offline Reinforcement Learning with Low-Rank Structure

Expert-Supervised Reinforcement Learning for Offline Policy Learning and Evaluation

Efficient and Sharp Off-Policy Evaluation in Robust Markov Decision Processes

Active Reinforcement Learning Strategies for Offline Policy Improvement

Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Sampling