Abstract:Lying on the heart of intelligent decision-making systems, how policy is represented and optimized is a fundamental problem. The root challenge in this problem is the large scale and the high complexity of policy space, which exacerbates the difficulty of policy learning especially in real-world scenarios. Towards a desirable surrogate policy space, recently policy representation in a low-dimensional latent space has shown its potential in improving both the evaluation and optimization of policy. The key question involved in these studies is by what criterion we should abstract the policy space for desired compression and generalization. However, both the theory on policy abstraction and the methodology on policy representation learning are less studied in the literature. In this work, we make very first efforts to fill up the vacancy. First, we propose a unified policy abstraction theory, containing three types of policy abstraction associated to policy features at different levels. Then, we generalize them to three policy metrics that quantify the distance (i.e., similarity) of policies, for more convenient use in learning policy representation. Further, we propose a policy representation learning approach based on deep metric learning. For the empirical study, we investigate the efficacy of the proposed policy metrics and representations, in characterizing policy difference and conveying policy generalization respectively. Our experiments are conducted in both policy optimization and evaluation problems, containing trust-region policy optimization (TRPO), diversity-guided evolution strategy (DGES) and off-policy evaluation (OPE). Somewhat naturally, the experimental results indicate that there is no a universally optimal abstraction for all downstream learning problems; while the influence-irrelevance policy abstraction can be a generally preferred choice.

Off-policy Evaluation with Deeply-abstracted States

Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Sampling

Off-Policy Evaluation via Off-Policy Classification

Off-policy Evaluation in Infinite-Horizon Reinforcement Learning with Latent Confounders

Causal Deepsets for Off-policy Evaluation under Spatial or Spatio-temporal Interferences

Off-policy evaluation beyond overlap: partial identification through smoothness

Off-policy Evaluation in Doubly Inhomogeneous Environments

OPERA: Automatic Offline Policy Evaluation with Re-weighted Aggregates of Multiple Estimators

Quantile Off-Policy Evaluation via Deep Conditional Generative Learning

State-Action Similarity-Based Representations for Off-Policy Evaluation

Off-Policy Evaluation in Partially Observable Environments

Offline Policy Evaluation in Large Action Spaces Via Outcome-Oriented Action Grouping

Off-Policy Evaluation for Large Action Spaces via Conjunct Effect Modeling

A Maximum-Entropy Approach to Off-Policy Evaluation in Average-Reward MDPs.

Distributional Off-policy Evaluation with Bellman Residual Minimization

Future-Dependent Value-Based Off-Policy Evaluation in POMDPs

$Δ\text{-}{\rm OPE}$: Off-Policy Estimation with Pairs of Policies

Offline Policy Evaluation for Reinforcement Learning with Adaptively Collected Data

Towards A Unified Policy Abstraction Theory and Representation Learning Approach in Markov Decision Processes

Off-Policy Evaluation Using Information Borrowing and Context-Based Switching