Abstract:We study sample efficient reinforcement learning (RL) under the general framework of interactive decision making, which includes Markov decision process (MDP), partially observable Markov decision process (POMDP), and predictive state representation (PSR) as special cases. Toward finding the minimum assumption that empowers sample efficient learning, we propose a novel complexity measure, generalized eluder coefficient (GEC), which characterizes the fundamental tradeoff between exploration and exploitation in online interactive decision making. In specific, GEC captures the hardness of exploration by comparing the error of predicting the performance of the updated policy with the in-sample training error evaluated on the historical data. We show that RL problems with low GEC form a remarkably rich class, which subsumes low Bellman eluder dimension problems, bilinear class, low witness rank problems, PO-bilinear class, and generalized regular PSR, where generalized regular PSR, a new tractable PSR class identified by us, includes nearly all known tractable POMDPs and PSRs. Furthermore, in terms of algorithm design, we propose a generic posterior sampling algorithm, which can be implemented in both model-free and model-based fashion, under both fully observable and partially observable settings. The proposed algorithm modifies the standard posterior sampling algorithm in two aspects: (i) we use an optimistic prior distribution that biases towards hypotheses with higher values and (ii) a loglikelihood function is set to be the empirical loss evaluated on the historical data, where the choice of loss function supports both model-free and model-based learning. We prove that the proposed algorithm is sample efficient by establishing a sublinear regret upper bound in terms of GEC. In summary, we provide a new and unified understanding of both fully observable and partially observable RL.

Exploration is Harder than Prediction: Cryptographically Separating Reinforcement Learning from Supervised Learning

Provably Efficient Exploration for Reinforcement Learning Using Unsupervised Learning

Provably Efficient Exploration for RL with Unsupervised Learning

A Survey on Explainable Reinforcement Learning: Concepts, Algorithms, Challenges

Fundamental Limits of Reinforcement Learning in Environment with Endogeneous and Exogeneous Uncertainty

What makes math problems hard for reinforcement learning: a case study

Provable Partially Observable Reinforcement Learning with Privileged Information

Provably Efficient UCB-type Algorithms For Learning Predictive State Representations

Distributionally Robust Reinforcement Learning with Interactive Data Collection: Fundamental Hardness and Near-Optimal Algorithm

Explore and Control with Adversarial Surprise

Beyond Optimism: Exploration With Partially Observable Rewards

On Reinforcement Learning with Adversarial Corruption and Its Application to Block MDP

When Your AIs Deceive You: Challenges of Partial Observability in Reinforcement Learning from Human Feedback

An Exponential Lower Bound for Linearly-Realizable MDPs with Constant Suboptimality Gap

Provably Efficient Exploration in Quantum Reinforcement Learning with Logarithmic Worst-Case Regret

Reinforcement Learning with Exogenous States and Rewards

Preserving the Privacy of Reward Functions in MDPs through Deception

GEC: A Unified Framework for Interactive Decision Making in MDP, POMDP, and Beyond

Reinforcement Learning with Probabilistically Complete Exploration

Regret Bounds and Reinforcement Learning Exploration of EXP-based Algorithms

Is Long Horizon Rl More Difficult Than Short Horizon RL?