Abstract:In this work, we consider the problem of model selection for deep reinforcement learning (RL) in real-world environments. Typically, the performance of deep RL algorithms is evaluated via on-policy interactions with the target environment. However, comparing models in a real-world environment for the purposes of early stopping or hyperparameter tuning is costly and often practically infeasible. This leads us to examine off-policy policy evaluation (OPE) in such settings. We focus on OPE for value-based methods, which are of particular interest in deep RL, with applications like robotics, where off-policy algorithms based on Q-function estimation can often attain better sample complexity than direct policy optimization. Existing OPE metrics either rely on a model of the environment, or the use of importance sampling (IS) to correct for the data being off-policy. However, for high-dimensional observations, such as images, models of the environment can be difficult to fit and value-based methods can make IS hard to use or even ill-conditioned, especially when dealing with continuous action spaces. In this paper, we focus on the specific case of MDPs with continuous action spaces and sparse binary rewards, which is representative of many important real-world applications. We propose an alternative metric that relies on neither models nor IS, by framing OPE as a positive-unlabeled (PU) classification problem with the Q-function as the decision function. We experimentally show that this metric outperforms baselines on a number of tasks. Most importantly, it can reliably predict the relative performance of different policies in a number of generalization scenarios, including the transfer to the real-world of policies trained in simulation for an image-based robotic manipulation task.

Cross-Validated Off-Policy Evaluation

Off-Policy Evaluation from Logged Human Feedback

Off-Policy Evaluation with Out-of-Sample Guarantees

Automated Off-Policy Estimator Selection via Supervised Learning

Off-Policy Evaluation via Off-Policy Classification

More Efficient Off-Policy Evaluation through Regularized Targeted Learning

Concept-driven Off Policy Evaluation

Is Cross-Validation the Gold Standard to Evaluate Model Performance?

Performance-Estimation Properties of Cross-Validation-Based Protocols with Simultaneous Hyper-Parameter Optimization

When is Off-Policy Evaluation (Reward Modeling) Useful in Contextual Bandits? A Data-Centric Perspective

Pessimistic Off-Policy Optimization for Learning to Rank

Off-Policy Evaluation in Partially Observable Environments

Off-Policy Evaluation for Large Action Spaces via Policy Convolution

Off-policy Evaluation in Infinite-Horizon Reinforcement Learning with Latent Confounders

Don't Waste Your Time: Early Stopping Cross-Validation

Optimal Mixture Weights for Off-Policy Evaluation with Multiple Behavior Policies

A survey of cross-validation procedures for model selection

Minimax Off-Policy Evaluation for Multi-Armed Bandits

Efficient Evaluation of Natural Stochastic Policies in Offline Reinforcement Learning

Conformal Off-Policy Evaluation in Markov Decision Processes

Sample-efficient Nonstationary Policy Evaluation for Contextual Bandits