Abstract:The modern paradigm in machine learning involves pre-training on diverse data, followed by task-specific fine-tuning. In reinforcement learning (RL), this translates to learning via offline RL on a diverse historical dataset, followed by rapid online RL fine-tuning using interaction data. Most RL fine-tuning methods require continued training on offline data for stability and performance. However, this is undesirable because training on diverse offline data is slow and expensive for large datasets, and in principle, also limit the performance improvement possible because of constraints or pessimism on offline data. In this paper, we show that retaining offline data is unnecessary as long as we use a properly-designed online RL approach for fine-tuning offline RL initializations. To build this approach, we start by analyzing the role of retaining offline data in online fine-tuning. We find that continued training on offline data is mostly useful for preventing a sudden divergence in the value function at the onset of fine-tuning, caused by a distribution mismatch between the offline data and online rollouts. This divergence typically results in unlearning and forgetting the benefits of offline pre-training. Our approach, Warm-start RL (WSRL), mitigates the catastrophic forgetting of pre-trained initializations using a very simple idea. WSRL employs a warmup phase that seeds the online RL run with a very small number of rollouts from the pre-trained policy to do fast online RL. The data collected during warmup helps ``recalibrate'' the offline Q-function to the online distribution, allowing us to completely discard offline data without destabilizing the online RL fine-tuning. We show that WSRL is able to fine-tune without retaining any offline data, and is able to learn faster and attains higher performance than existing algorithms irrespective of whether they retain offline data or not.

Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning.

DROP: Conservative Model-based Optimization for Offline Reinforcement Learning

A Rank-Based Sampling Framework for Offline Reinforcement Learning

Strategically Conservative Q-Learning

Improving Offline-to-Online Reinforcement Learning with Q Conditioned State Entropy Exploration

Offline Reinforcement Learning with Implicit Q-Learning

Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbone

Mildly Conservative Q-Learning for Offline Reinforcement Learning

Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions

Efficient and Stable Offline-to-online Reinforcement Learning Via Continual Policy Revitalization

Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data

Offline Quantum Reinforcement Learning in a Conservative Manner

A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning

Counterfactual Conservative Q Learning for Offline Multi-agent Reinforcement Learning

Adaptive Behavior Cloning Regularization for Stable Offline-to-Online Reinforcement Learning

A Minimalist Approach to Offline Reinforcement Learning

Offline RL with No OOD Actions: In-Sample Learning Via Implicit Value Regularization

Leveraging Offline Data in Online Reinforcement Learning

Adaptable Conservative Q-Learning for Offline Reinforcement Learning.

Hybrid RL: Using Both Offline and Online Data Can Make RL Efficient

CLUE: Calibrated Latent Guidance for Offline Reinforcement Learning