Abstract:Offline pretraining with a static dataset followed by online fine-tuning (offline-to-online, or OtO) is a paradigm well matched to a real-world RL deployment process. In this scenario, we aim to find the best-performing policy within a limited budget of online interactions. Previous work in the OtO setting has focused on correcting for bias introduced by the policy-constraint mechanisms of offline RL algorithms. Such constraints keep the learned policy close to the behavior policy that collected the dataset, but we show this can unnecessarily limit policy performance if the behavior policy is far from optimal. Instead, we forgo constraints and frame OtO RL as an exploration problem that aims to maximize the benefit of online data-collection. We first study the major online RL exploration methods based on intrinsic rewards and UCB in the OtO setting, showing that intrinsic rewards add training instability through reward-function modification, and UCB methods are myopic and it is unclear which learned-component's ensemble to use for action selection. We then introduce an algorithm for planning to go out-of-distribution (PTGOOD) that avoids these issues. PTGOOD uses a non-myopic planning procedure that targets exploration in relatively high-reward regions of the state-action space unlikely to be visited by the behavior policy. By leveraging concepts from the Conditional Entropy Bottleneck, PTGOOD encourages data collected online to provide new information relevant to improving the final deployment policy without altering rewards. We show empirically in several continuous control tasks that PTGOOD significantly improves agent returns during online fine-tuning and avoids the suboptimal policy convergence that many of our baselines exhibit in several environments.

Uncertainty-based Bootstrapped Optimization for Offline Reinforcement Learning

Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning

DROP: Conservative Model-based Optimization for Offline Reinforcement Learning

Towards Robust Offline-to-Online Reinforcement Learning via Uncertainty and Smoothness

Uncertainty-Aware Data Augmentation for Offline Reinforcement Learning

UAC: Offline Reinforcement Learning with Uncertain Action Constraint

Uncertainty-aware Distributional Offline Reinforcement Learning

Model-Based Offline Reinforcement Learning with Uncertainty Estimation and Policy Constraint

Double Actors and Uncertainty-Weighted Critics for Offline Reinforcement Learning.

Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL

Behavior Proximal Policy Optimization

Diverse Randomized Value Functions: A Provably Pessimistic Approach for Offline Reinforcement Learning

Offline Reinforcement Learning with Policy Guidance and Uncertainty Estimation.

A Simple Unified Uncertainty-Guided Framework for Offline-to-Online Reinforcement Learning

Enhancing OOD Generalization in Offline Reinforcement Learning with Energy-Based Policy Optimization

Planning to Go Out-of-Distribution in Offline-to-Online Reinforcement Learning

OSCAR: OOD State-Conservative Offline Reinforcement Learning for Sequential Decision Making

MUSBO: Model-based Uncertainty Regularized and Sample Efficient Batch Optimization for Deployment Constrained Reinforcement Learning

Double pessimism is provably efficient for distributionally robust offline reinforcement learning: Generic algorithm and robust partial coverage

Towards Robust Off-Policy Learning for Runtime Uncertainty

DARL: Distance-Aware Uncertainty Estimation for Offline Reinforcement Learning.