Abstract:Offline (or batch) reinforcement learning (RL) algorithms seek to learn an optimal policy from a fixed dataset without active data collection. Based on the composition of the offline dataset, two main categories of methods are used: imitation learning which is suitable for expert datasets and vanilla offline RL which often requires uniform coverage datasets. From a practical standpoint, datasets often deviate from these two extremes and the exact data composition is usually unknown a priori. To bridge this gap, we present a new offline RL framework that smoothly interpolates between the two extremes of data composition, hence unifying imitation learning and vanilla offline RL. The new framework is centered around a weak version of the concentrability coefficient that measures the deviation from the behavior policy to the expert policy alone. Under this new framework, we further investigate the question on algorithm design: can one develop an algorithm that achieves a minimax optimal rate and also adapts to unknown data composition? To address this question, we consider a lower confidence bound (LCB) algorithm developed based on pessimism in the face of uncertainty in offline RL. We study finite-sample properties of LCB as well as information-theoretic limits in multi-armed bandits, contextual bandits, and Markov decision processes (MDPs). Our analysis reveals surprising facts about optimality rates. In particular, in all three settings, LCB achieves a faster rate of $1/N$ for nearly-expert datasets compared to the usual rate of $1/\sqrt{N}$ in offline RL, where $N$ is the number of samples in the batch dataset. In the case of contextual bandits with at least two contexts, we prove that LCB is adaptively optimal for the entire data composition range, achieving a smooth transition from imitation learning to offline RL. We further show that LCB is almost adaptively optimal in MDPs.

Leveraging (Biased) Information: Multi-armed Bandits with Offline Data

Robustly Improving Bandit Algorithms with Confounded and Selection Biased Offline Data: A Causal Approach

Leveraging Offline Data in Linear Latent Bandits

Robust Offline Policy Learning with Observational Data from Multiple Sources

Online Learning with Diverse User Preferences

Online Algorithms for the Multi-Armed Bandit Problem with Markovian Rewards

Optimal Best-Arm Identification in Bandits with Access to Offline Data

UCB algorithms for multi-armed bandits: Precise regret and adaptive inference

Offline Planning and Online Learning Under Recovering Rewards

Restless-UCB, an Efficient and Low-complexity Algorithm for Online Restless Bandits.

Online Learning for Auction Mechanism in Bandit Setting

Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism

A Framework for Adapting Offline Algorithms to Solve Combinatorial Multi-Armed Bandit Problems with Bandit Feedback

Auction-Based Combinatorial Multi-Armed Bandit Mechanisms with Strategic Arms

Bayesian Regret Minimization in Offline Bandits

Multi-Armed Bandit Strategies for Non-Stationary Reward Distributions and Delayed Feedback Processes

Learning with Guarantee via Constrained Multi-armed Bandit: Theory and Network Applications

Federated Combinatorial Multi-Agent Multi-Armed Bandits

Minimax Off-Policy Evaluation for Multi-Armed Bandits

Off-Policy Evaluation and Learning from Logged Bandit Feedback: Error Reduction via Surrogate Policy.

Combining Online Learning and Offline Learning for Contextual Bandits with Deficient Support