Abstract:Lying on the heart of intelligent decision-making systems, how policy is represented and optimized is a fundamental problem. The root challenge in this problem is the large scale and the high complexity of policy space, which exacerbates the difficulty of policy learning especially in real-world scenarios. Towards a desirable surrogate policy space, recently policy representation in a low-dimensional latent space has shown its potential in improving both the evaluation and optimization of policy. The key question involved in these studies is by what criterion we should abstract the policy space for desired compression and generalization. However, both the theory on policy abstraction and the methodology on policy representation learning are less studied in the literature. In this work, we make very first efforts to fill up the vacancy. First, we propose a unified policy abstraction theory, containing three types of policy abstraction associated to policy features at different levels. Then, we generalize them to three policy metrics that quantify the distance (i.e., similarity) of policies, for more convenient use in learning policy representation. Further, we propose a policy representation learning approach based on deep metric learning. For the empirical study, we investigate the efficacy of the proposed policy metrics and representations, in characterizing policy difference and conveying policy generalization respectively. Our experiments are conducted in both policy optimization and evaluation problems, containing trust-region policy optimization (TRPO), diversity-guided evolution strategy (DGES) and off-policy evaluation (OPE). Somewhat naturally, the experimental results indicate that there is no a universally optimal abstraction for all downstream learning problems; while the influence-irrelevance policy abstraction can be a generally preferred choice.

Policy optimization emerges from noisy representation learning

Representation-Driven Reinforcement Learning

No Representation, No Trust: Connecting Representation, Collapse, and Trust Issues in PPO

Policy Optimization in a Noisy Neighborhood: On Return Landscapes in Continuous Control

Overcoming Reward Overoptimization via Adversarial Policy Optimization with Lightweight Uncertainty Estimation

Reward Model Ensembles Help Mitigate Overoptimization

Neuron-level prediction and noise can implement flexible reward-seeking behavior

Scheduled Policy Optimization for Natural Language Communication with Intelligent Agents.

How Do You Act? An Empirical Study to Understand Behavior of Deep Reinforcement Learning Agents

PolicyBoost: Functional Policy Gradient with Ranking-based Reward Objective

Examining Policy Entropy of Reinforcement Learning Agents for Personalization Tasks

Three Pathways to Neurosymbolic Reinforcement Learning with Interpretable Model and Policy Networks

Model-based Policy Optimization using Symbolic World Model

Iterative Amortized Policy Optimization

Towards A Unified Policy Abstraction Theory and Representation Learning Approach in Markov Decision Processes

Entropy-Regularized Token-Level Policy Optimization for Language Agent Reinforcement

OMPO: A Unified Framework for RL under Policy and Dynamics Shifts

Adversarial Policy Optimization in Deep Reinforcement Learning

Model-Based Offline Weighted Policy Optimization (Student Abstract)

Optimizing Interpretable Decision Tree Policies for Reinforcement Learning