Abstract:Recent theoretical work studies sample-efficient reinforcement learning (RL) extensively in two settings: learning interactively in the environment (online RL), or learning from an offline dataset (offline RL). However, existing algorithms and theories for learning near-optimal policies in these two settings are rather different and disconnected. Towards bridging this gap, this paper initiates the theoretical study of policy finetuning, that is, online RL where the learner has additional access to a "reference policy" $\mu$ close to the optimal policy $\pi_\star$ in a certain sense. We consider the policy finetuning problem in episodic Markov Decision Processes (MDPs) with $S$ states, $A$ actions, and horizon length $H$. We first design a sharp offline reduction algorithm -- which simply executes $\mu$ and runs offline policy optimization on the collected dataset -- that finds an $\varepsilon$ near-optimal policy within $\widetilde{O}(H^3SC^\star/\varepsilon^2)$ episodes, where $C^\star$ is the single-policy concentrability coefficient between $\mu$ and $\pi_\star$. This offline result is the first that matches the sample complexity lower bound in this setting, and resolves a recent open question in offline RL. We then establish an $\Omega(H^3S\min\{C^\star, A\}/\varepsilon^2)$ sample complexity lower bound for any policy finetuning algorithm, including those that can adaptively explore the environment. This implies that -- perhaps surprisingly -- the optimal policy finetuning algorithm is either offline reduction or a purely online RL algorithm that does not use $\mu$. Finally, we design a new hybrid offline/online algorithm for policy finetuning that achieves better sample complexity than both vanilla offline reduction and purely online RL algorithms, in a relaxed setting where $\mu$ only satisfies concentrability partially up to a certain time step.

On Sampling Efficiency Optimization in Constrained Reinforcement Learning*

On Efficient Sampling for Reinforcement Learning with Multiple Constraints*

On Efficient Sampling in Offline Reinforcement Learning

Successive Convex Approximation Based Off-Policy Optimization for Constrained Reinforcement Learning

On Efficient Sampling in Supervisory Reinforcement Learning

A Rank-Based Sampling Framework for Offline Reinforcement Learning

Optimal Control for Constrained Discrete-Time Nonlinear Systems Based on Safe Reinforcement Learning.

Off-Policy RL Algorithms Can be Sample-Efficient for Continuous Control via Sample Multiple Reuse

An OCBA-Based Method for Efficient Sample Collection in Reinforcement Learning

Optimistic Sampling Strategy for Data-Efficient Reinforcement Learning

Robust On-Policy Sampling for Data-Efficient Policy Evaluation in Reinforcement Learning

Constrained stochastic optimal control with learned importance sampling: A path integral approach

Estimation and Control Using Sampling-Based Bayesian Reinforcement Learning

Provably Efficient Exploration in Inverse Constrained Reinforcement Learning

Enhancing Efficiency of Safe Reinforcement Learning via Sample Manipulation

Sample-Efficient Reinforcement Learning Is Feasible for Linearly Realizable MDPs with Limited Revisiting

Constraints Penalized Q-learning for Safe Offline Reinforcement Learning.

Nearly Minimax Optimal Reward-free Reinforcement Learning.

Off-Policy Risk-Sensitive Reinforcement Learning-Based Constrained Robust Optimal Control

Sample Efficient Reinforcement Learning via Low-Rank Matrix Estimation

Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning