Abstract:Offline Reinforcement Learning (RL), which operates solely on static datasets without further interactions with the environment, provides an appealing alternative to learning a safe and promising control policy. The prevailing methods typically learn a conservative policy to mitigate the problem of Q-value overestimation, but it is prone to overdo it, leading to an overly conservative policy. Moreover, they optimize all samples equally with fixed constraints, lacking the nuanced ability to control conservative levels in a fine-grained manner. Consequently, this limitation results in a performance decline. To address the above two challenges in a united way, we propose a framework, Adaptive Conservative Level in Q-Learning (ACL-QL), which limits the Q-values in a mild range and enables adaptive control on the conservative level over each state-action pair, i.e., lifting the Q-values more for good transitions and less for bad transitions. We theoretically analyze the conditions under which the conservative level of the learned Q-function can be limited in a mild range and how to optimize each transition adaptively. Motivated by the theoretical analysis, we propose a novel algorithm, ACL-QL, which uses two learnable adaptive weight functions to control the conservative level over each transition. Subsequently, we design a monotonicity loss and surrogate losses to train the adaptive weight functions, Q-function, and policy network alternatively. We evaluate ACL-QL on the commonly used D4RL benchmark and conduct extensive ablation studies to illustrate the effectiveness and state-of-the-art performance compared to existing offline DRL baselines.

Judgmentally adjusted Q-values based on Q-ensemble for offline reinforcement learning

A Rank-Based Sampling Framework for Offline Reinforcement Learning

Q-Distribution guided Q-learning for offline reinforcement learning: Uncertainty penalized Q-value via consistency model

A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning

Mildly Conservative Q-Learning for Offline Reinforcement Learning

Adaptable Conservative Q-Learning for Offline Reinforcement Learning.

Improving Offline-to-Online Reinforcement Learning with Q Conditioned State Entropy Exploration

Boosting Offline Reinforcement Learning via Data Rebalancing

Offline Reinforcement Learning with Imbalanced Datasets

Interpretable performance analysis towards offline reinforcement learning: A dataset perspective

Adaptive pessimism via target Q-value for offline reinforcement learning

Constrained Policy Optimization with Explicit Behavior Density for Offline Reinforcement Learning

Offline RL with No OOD Actions: In-Sample Learning Via Implicit Value Regularization

ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learning

ENOTO: Improving Offline-to-Online Reinforcement Learning with Q-Ensembles

Entropy-regularized Diffusion Policy with Q-Ensembles for Offline Reinforcement Learning

Constraints Penalized Q-learning for Safe Offline Reinforcement Learning.

Conservative In-Distribution Q-Learning for Offline Reinforcement Learning

ACQL: An Adaptive Conservative Q-Learning Framework for Offline Reinforcement Learning

Offline Reinforcement Learning with On-Policy Q-Function Regularization

Offline Reinforcement Learning With Behavior Value Regularization