Abstract:Offline pretraining with a static dataset followed by online fine-tuning (offline-to-online, or OtO) is a paradigm well matched to a real-world RL deployment process. In this scenario, we aim to find the best-performing policy within a limited budget of online interactions. Previous work in the OtO setting has focused on correcting for bias introduced by the policy-constraint mechanisms of offline RL algorithms. Such constraints keep the learned policy close to the behavior policy that collected the dataset, but we show this can unnecessarily limit policy performance if the behavior policy is far from optimal. Instead, we forgo constraints and frame OtO RL as an exploration problem that aims to maximize the benefit of online data-collection. We first study the major online RL exploration methods based on intrinsic rewards and UCB in the OtO setting, showing that intrinsic rewards add training instability through reward-function modification, and UCB methods are myopic and it is unclear which learned-component's ensemble to use for action selection. We then introduce an algorithm for planning to go out-of-distribution (PTGOOD) that avoids these issues. PTGOOD uses a non-myopic planning procedure that targets exploration in relatively high-reward regions of the state-action space unlikely to be visited by the behavior policy. By leveraging concepts from the Conditional Entropy Bottleneck, PTGOOD encourages data collected online to provide new information relevant to improving the final deployment policy without altering rewards. We show empirically in several continuous control tasks that PTGOOD significantly improves agent returns during online fine-tuning and avoids the suboptimal policy convergence that many of our baselines exhibit in several environments.

Offline Reinforcement Learning Via Optimal Transport and Improved Performance Difference Theorem

Behavior Proximal Policy Optimization

Beyond Reward: Offline Preference-guided Policy Optimization

Rethinking Optimal Transport in Offline Reinforcement Learning

Optimal Transport for Offline Imitation Learning

Boosting Offline Reinforcement Learning via Data Rebalancing

Align Your Intents: Offline Imitation Learning via Optimal Transport

Preferred-Action-Optimized Diffusion Policies for Offline Reinforcement Learning

Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL

Offline Reinforcement Learning for Wireless Network Optimization with Mixture Datasets

Interpretable performance analysis towards offline reinforcement learning: A dataset perspective

Offline Reinforcement Learning with Imbalanced Datasets

Planning to Go Out-of-Distribution in Offline-to-Online Reinforcement Learning

A2PO: Towards Effective Offline Reinforcement Learning from an Advantage-aware Perspective

Robust Offline Reinforcement Learning from Low-Quality Data

Augmenting Offline RL with Unlabeled Data

MOPO: Model-based Offline Policy Optimization

Adaptive Policy Learning for Offline-to-Online Reinforcement Learning

Offline RL With Realistic Datasets: Heteroskedasticity and Support Constraints

Harnessing Mixed Offline Reinforcement Learning Datasets via Trajectory Weighting

Enhancing OOD Generalization in Offline Reinforcement Learning with Energy-Based Policy Optimization