Abstract:Offline reinforcement learning requires reconciling two conflicting aims: learning a policy that improves over the behavior policy that collected the dataset, while at the same time minimizing the deviation from the behavior policy so as to avoid errors due to distributional shift. This trade-off is critical, because most current offline reinforcement learning methods need to query the value of unseen actions during training to improve the policy, and therefore need to either constrain these actions to be in-distribution, or else regularize their values. We propose an offline RL method that never needs to evaluate actions outside of the dataset, but still enables the learned policy to improve substantially over the best behavior in the data through generalization. The main insight in our work is that, instead of evaluating unseen actions from the latest policy, we can approximate the policy improvement step implicitly by treating the state value function as a random variable, with randomness determined by the action (while still integrating over the dynamics to avoid excessive optimism), and then taking a state conditional upper expectile of this random variable to estimate the value of the best actions in that state. This leverages the generalization capacity of the function approximator to estimate the value of the best available action at a given state without ever directly querying a Q-function with this unseen action. Our algorithm alternates between fitting this upper expectile value function and backing it up into a Q-function. Then, we extract the policy via advantage-weighted behavioral cloning. We dub our method implicit Q-learning (IQL). IQL demonstrates the state-of-the-art performance on D4RL, a standard benchmark for offline reinforcement learning. We also demonstrate that IQL achieves strong performance fine-tuning using online interaction after offline initialization.

Adaptive $Q$-Aid for Conditional Supervised Learning in Offline Reinforcement Learning

Successive Convex Approximation Based Off-Policy Optimization for Constrained Reinforcement Learning

ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learning

ACQL: An Adaptive Conservative Q-Learning Framework for Offline Reinforcement Learning

Q-value Regularized Transformer for Offline Reinforcement Learning

Adaptable Conservative Q-Learning for Offline Reinforcement Learning.

Improving Offline-to-Online Reinforcement Learning with Q Conditioned State Entropy Exploration

Strategically Conservative Q-Learning

Constraints Penalized Q-learning for Safe Offline Reinforcement Learning.

SaFormer: A Conditional Sequence Modeling Approach to Offline Safe Reinforcement Learning

State-Constrained Offline Reinforcement Learning

Offline Reinforcement Learning with On-Policy Q-Function Regularization

Mildly Conservative Q-Learning for Offline Reinforcement Learning

Offline Reinforcement Learning with Implicit Q-Learning

Q-value Regularized Decision ConvFormer for Offline Reinforcement Learning

Free from Bellman Completeness: Trajectory Stitching via Model-based Return-conditioned Supervised Learning

Offline RL With Realistic Datasets: Heteroskedasticity and Support Constraints

Offline Quantum Reinforcement Learning in a Conservative Manner

Robust Offline Actor-Critic with On-Policy Regularized Policy Evaluation

Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning

Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning.