Abstract:A fundamental question in the theory of reinforcement learning is: suppose the optimal $Q$-function lies in the linear span of a given $d$ dimensional feature mapping, is sample-efficient reinforcement learning (RL) possible? The recent and remarkable result of Weisz et al. (2020) resolved this question in the negative, providing an exponential (in $d$) sample size lower bound, which holds even if the agent has access to a generative model of the environment. One may hope that this information theoretic barrier for RL can be circumvented by further supposing an even more favorable assumption: there exists a \emph{constant suboptimality gap} between the optimal $Q$-value of the best action and that of the second-best action (for all states). The hope is that having a large suboptimality gap would permit easier identification of optimal actions themselves, thus making the problem tractable; indeed, provided the agent has access to a generative model, sample-efficient RL is in fact possible with the addition of this more favorable assumption. This work focuses on this question in the standard online reinforcement learning setting, where our main result resolves this question in the negative: our hardness result shows that an exponential sample complexity lower bound still holds even if a constant suboptimality gap is assumed in addition to having a linearly realizable optimal $Q$-function. Perhaps surprisingly, this implies an exponential separation between the online RL setting and the generative model setting. Complementing our negative hardness result, we give two positive results showing that provably sample-efficient RL is possible either under an additional low-variance assumption or under a novel hypercontractivity assumption (both implicitly place stronger conditions on the underlying dynamics model).

Settling the Sample Complexity of Online Reinforcement Learning

Breaking the sample complexity barrier to regret-optimal model-free reinforcement learning

Settling the Sample Complexity of Model-Based Offline Reinforcement Learning

A Rank-Based Sampling Framework for Offline Reinforcement Learning

Achieving the Asymptotically Optimal Sample Complexity of Offline Reinforcement Learning: A DRO-Based Approach

Hybrid Reinforcement Learning Breaks Sample Size Barriers in Linear MDPs

Efficient Online Reinforcement Learning with Offline Data

On Sample-Efficient Offline Reinforcement Learning: Data Diversity, Posterior Sampling, and Beyond

Efficient Deep Reinforcement Learning Requires Regulating Overfitting

Offline RL with Observation Histories: Analyzing and Improving Sample Complexity

Online Sub-Sampling for Reinforcement Learning with General Function Approximation

An Exponential Lower Bound for Linearly-Realizable MDPs with Constant Suboptimality Gap

Sample-Efficient Reinforcement Learning Is Feasible for Linearly Realizable MDPs with Limited Revisiting

Sample Efficient Reinforcement Learning with Partial Dynamics Knowledge

Regret-Optimal Model-Free Reinforcement Learning for Discounted MDPs with Short Burn-In Time

Model-based RL as a Minimalist Approach to Horizon-Free and Second-Order Bounds

A Minimalist Approach to Offline Reinforcement Learning

Settling the Horizon-Dependence of Sample Complexity in Reinforcement Learning

Actions Speak What You Want: Provably Sample-Efficient Reinforcement Learning of the Quantal Stackelberg Equilibrium from Strategic Feedbacks

Sample Complexity of Offline Distributionally Robust Linear Markov Decision Processes

Efficient Constrained Regret Minimization