Abstract:This study focuses on the topic of offline preference-based reinforcement learning (PbRL), a variant of conventional reinforcement learning that dispenses with the need for online interaction or specification of reward functions. Instead, the agent is provided with fixed offline trajectories and human preferences between pairs of trajectories to extract the dynamics and task information, respectively. Since the dynamics and task information are orthogonal, a naive approach would involve using preference-based reward learning followed by an off-the-shelf offline RL algorithm. However, this requires the separate learning of a scalar reward function, which is assumed to be an information bottleneck of the learning process. To address this issue, we propose the offline preference-guided policy optimization (OPPO) paradigm, which models offline trajectories and preferences in a one-step process, eliminating the need for separately learning a reward function. OPPO achieves this by introducing an offline hindsight information matching objective for optimizing a contextual policy and a preference modeling objective for finding the optimal context. OPPO further integrates a well-performing decision policy by optimizing the two objectives iteratively. Our empirical results demonstrate that OPPO effectively models offline preferences and outperforms prior competing baselines, including offline RL algorithms performed over either true or pseudo reward function specifications. Our code is available on the project website: https://sites.google.com/view/oppo-icml-2023 .

Identifying drug-induced lung injury in a patient with rheumatoid arthritis

Behavior Proximal Policy Optimization

Beyond Reward: Offline Preference-guided Policy Optimization

DROP: Conservative Model-based Optimization for Offline Reinforcement Learning

Offline Reinforcement Learning with Soft Behavior Regularization

Supported Value Regularization for Offline Reinforcement Learning

Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL

Mildly Conservative Q-Learning for Offline Reinforcement Learning

Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learning

Conservative Bayesian Model-Based Value Expansion for Offline Policy Optimization

A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning

Mildly Constrained Evaluation Policy for Offline Reinforcement Learning

Risk-Averse Offline Reinforcement Learning

Design from Policies: Conservative Test-Time Adaptation for Offline Policy Optimization

Efficient Offline Reinforcement Learning: The Critic is Critical

De-Pessimism Offline Reinforcement Learning via Value Compensation

Robust Offline Reinforcement Learning from Low-Quality Data

Actor-Critic Alignment for Offline-to-Online Reinforcement Learning

Offline RL with No OOD Actions: In-Sample Learning Via Implicit Value Regularization

In-sample Actor Critic for Offline Reinforcement Learning

VOCE: Variational Optimization with Conservative Estimation for Offline Safe Reinforcement Learning.