Abstract:Offline reinforcement learning (RL) promises the ability to learn effective policies solely using existing, static datasets, without any costly online interaction. To do so, offline RL methods must handle distributional shift between the dataset and the learned policy. The most common approach is to learn conservative, or lower-bound, value functions, which underestimate the return of out-of-distribution (OOD) actions. However, such methods exhibit one notable drawback: policies optimized on such value functions can only behave according to a fixed, possibly suboptimal, degree of conservatism. However, this can be alleviated if we instead are able to learn policies for varying degrees of conservatism at training time and devise a method to dynamically choose one of them during evaluation. To do so, in this work, we propose learning value functions that additionally condition on the degree of conservatism, which we dub confidence-conditioned value functions. We derive a new form of a Bellman backup that simultaneously learns Q-values for any degree of confidence with high probability. By conditioning on confidence, our value functions enable adaptive strategies during online evaluation by controlling for confidence level using the history of observations thus far. This approach can be implemented in practice by conditioning the Q-function from existing conservative algorithms on the confidence.We theoretically show that our learned value functions produce conservative estimates of the true value at any desired confidence. Finally, we empirically show that our algorithm outperforms existing conservative offline RL algorithms on multiple discrete control domains.

Adaptable Conservative Q-Learning for Offline Reinforcement Learning.

DROP: Conservative Model-based Optimization for Offline Reinforcement Learning

Design from Policies: Conservative Test-Time Adaptation for Offline Policy Optimization

Mildly Conservative Q-Learning for Offline Reinforcement Learning

Strategically Conservative Q-Learning

Counterfactual Conservative Q Learning for Offline Multi-agent Reinforcement Learning

Q-Distribution guided Q-learning for offline reinforcement learning: Uncertainty penalized Q-value via consistency model

Robust Offline Reinforcement Learning from Low-Quality Data

Adaptive pessimism via target Q-value for offline reinforcement learning

DCE: Offline Reinforcement Learning with Double Conservative Estimates

Efficient Offline Reinforcement Learning With Relaxed Conservatism

Constraints Penalized Q-learning for Safe Offline Reinforcement Learning.

UAC: Offline Reinforcement Learning with Uncertain Action Constraint

Confidence-Conditioned Value Functions for Offline Reinforcement Learning

Offline Quantum Reinforcement Learning in a Conservative Manner

A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning

CDSA: Conservative Denoising Score-based Algorithm for Offline Reinforcement Learning

Judgmentally adjusted Q-values based on Q-ensemble for offline reinforcement learning

Offline Decentralized Multi-Agent Reinforcement Learning

Robust Offline Reinforcement Learning with Gradient Penalty and Constraint Relaxation

Constrained Policy Optimization with Explicit Behavior Density for Offline Reinforcement Learning