Abstract:Lying on the heart of intelligent decision-making systems, how policy is represented and optimized is a fundamental problem. The root challenge in this problem is the large scale and the high complexity of policy space, which exacerbates the difficulty of policy learning especially in real-world scenarios. Towards a desirable surrogate policy space, recently policy representation in a low-dimensional latent space has shown its potential in improving both the evaluation and optimization of policy. The key question involved in these studies is by what criterion we should abstract the policy space for desired compression and generalization. However, both the theory on policy abstraction and the methodology on policy representation learning are less studied in the literature. In this work, we make very first efforts to fill up the vacancy. First, we propose a unified policy abstraction theory, containing three types of policy abstraction associated to policy features at different levels. Then, we generalize them to three policy metrics that quantify the distance (i.e., similarity) of policies, for more convenient use in learning policy representation. Further, we propose a policy representation learning approach based on deep metric learning. For the empirical study, we investigate the efficacy of the proposed policy metrics and representations, in characterizing policy difference and conveying policy generalization respectively. Our experiments are conducted in both policy optimization and evaluation problems, containing trust-region policy optimization (TRPO), diversity-guided evolution strategy (DGES) and off-policy evaluation (OPE). Somewhat naturally, the experimental results indicate that there is no a universally optimal abstraction for all downstream learning problems; while the influence-irrelevance policy abstraction can be a generally preferred choice.

UPPAM: A Unified Pre-training Architecture for Political Actor Modeling based on Language

Political Actor Agent: Simulating Legislative System for Roll Call Votes Prediction with Large Language Models

PAR: Political Actor Representation Learning with Social Context and Expert Knowledge

Unifying Local and Global Knowledge: Empowering Large Language Models As Political Experts with Knowledge Graphs

Thoroughly Modeling Multi-domain Pre-trained Recommendation as Language

Unleashing the Power of Language Models in Text-Attributed Graph.

Unleashing the Power of Language Models in Text-Attributed Graph

UnifiedMLLM: Enabling Unified Representation for Multi-modal Multi-tasks With Large Language Model

Unified Language Model Pre-training for Natural Language Understanding and Generation

UniCorn: A Unified Contrastive Learning Approach for Multi-view Molecular Representation Learning

Towards A Unified Policy Abstraction Theory and Representation Learning Approach in Markov Decision Processes

Legislator Representation Learning with Social Context and Expert Knowledge

Human Language Modeling

Uni3DL: Unified Model for 3D and Language Understanding

Unifying Structure Reasoning and Language Model Pre-training for Complex Reasoning

Cooperative Policy Learning with Pre-trained Heterogeneous Observation Representations

Unifying Structure Reasoning and Language Pre-Training for Complex Reasoning Tasks

Opponent portrait for multiagent reinforcement learning in competitive environment

Knowing Where and What: Unified Word Block Pretraining for Document Understanding

Pre-training Multi-party Dialogue Models with Latent Discourse Inference.

UNILMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training