Abstract:This paper studies the fundamental limits of reinforcement learning (RL) in the challenging \emph{partially observable} setting. While it is well-established that learning in Partially Observable Markov Decision Processes (POMDPs) requires exponentially many samples in the worst case, a surge of recent work shows that polynomial sample complexities are achievable under the \emph{revealing condition} -- A natural condition that requires the observables to reveal some information about the unobserved latent states. However, the fundamental limits for learning in revealing POMDPs are much less understood, with existing lower bounds being rather preliminary and having substantial gaps from the current best upper bounds. We establish strong PAC and regret lower bounds for learning in revealing POMDPs. Our lower bounds scale polynomially in all relevant problem parameters in a multiplicative fashion, and achieve significantly smaller gaps against the current best upper bounds, providing a solid starting point for future studies. In particular, for \emph{multi-step} revealing POMDPs, we show that (1) the latent state-space dependence is at least $\Omega(S^{1.5})$ in the PAC sample complexity, which is notably harder than the $\widetilde{\Theta}(S)$ scaling for fully-observable MDPs; (2) Any polynomial sublinear regret is at least $\Omega(T^{2/3})$, suggesting its fundamental difference from the \emph{single-step} case where $\widetilde{O}(\sqrt{T})$ regret is achievable. Technically, our hard instance construction adapts techniques in \emph{distribution testing}, which is new to the RL literature and may be of independent interest.

Sublinear Regret for Learning POMDPs

Efficient Learning of POMDPs with Known Observation Model in Average-Reward Setting

Logarithmic Regret Bounds for Continuous-Time Average-Reward Markov Decision Processes

Lower Bounds for Learning in Revealing POMDPs

Horizon-Free and Variance-Dependent Reinforcement Learning for Latent Markov Decision Processes

Posterior Sampling-based Online Learning for Episodic POMDPs

Learning in Observable POMDPs, without Computationally Intractable Oracles

√N-Regret for Learning in Markov Decision Processes with Function Approximation and Low Bellman Rank.

Sample-Efficient Learning of POMDPs with Multiple Observations In Hindsight

Online Reinforcement Learning in Markov Decision Process Using Linear Programming

Provably Efficient Partially Observable Risk-Sensitive Reinforcement Learning with Hindsight Observation

$\Sqrt{n}$-Regret for Learning in Markov Decision Processes with Function Approximation and Low Bellman Rank

Refined Regret for Adversarial MDPs with Linear Function Approximation

Horizon-Free Regret for Linear Markov Decision Processes

Square-root regret bounds for continuous-time episodic Markov decision processes

Online Markov Decision Processes with Non-Oblivious Strategic Adversary

Dynamic Regret of Online Markov Decision Processes

Value-Biased Maximum Likelihood Estimation for Model-based Reinforcement Learning in Discounted Linear MDPs

Provably Efficient Reinforcement Learning for Infinite-Horizon Average-Reward Linear MDPs