Abstract:Partial Observability -- where agents can only observe partial information about the true underlying state of the system -- is ubiquitous in real-world applications of Reinforcement Learning (RL). Theoretically, learning a near-optimal policy under partial observability is known to be hard in the worst case due to an exponential sample complexity lower bound. Recent work has identified several tractable subclasses that are learnable with polynomial samples, such as Partially Observable Markov Decision Processes (POMDPs) with certain revealing or decodability conditions. However, this line of research is still in its infancy, where (1) unified structural conditions enabling sample-efficient learning are lacking; (2) existing sample complexities for known tractable subclasses are far from sharp; and (3) fewer sample-efficient algorithms are available than in fully observable RL. This paper advances all three aspects above for Partially Observable RL in the general setting of Predictive State Representations (PSRs). First, we propose a natural and unified structural condition for PSRs called \emph{B-stability}. B-stable PSRs encompasses the vast majority of known tractable subclasses such as weakly revealing POMDPs, low-rank future-sufficient POMDPs, decodable POMDPs, and regular PSRs. Next, we show that any B-stable PSR can be learned with polynomial samples in relevant problem parameters. When instantiated in the aforementioned subclasses, our sample complexities improve substantially over the current best ones. Finally, our results are achieved by three algorithms simultaneously: Optimistic Maximum Likelihood Estimation, Estimation-to-Decisions, and Model-Based Optimistic Posterior Sampling. The latter two algorithms are new for sample-efficient learning of POMDPs/PSRs.

Survey of Predictive State Representations

Survey on predictive representations of state

Predictive Models in Software Engineering: Challenges and Opportunities

Hilbert Space Embeddings of Predictive State Representations

Extended State Space Predictive Control for a Class of Nonlinear Systems

Provably Efficient UCB-type Algorithms For Learning Predictive State Representations

Predictive-State Decoders: Encoding the Future into Recurrent Networks

Partially Observable RL with B-Stability: Unified Structural Condition and Sharp Sample-Efficient Algorithms

Probabilistic State-Dependent Grammars for Plan Recognition

Status quo and prospect of power system state estimation

Predictive Position Control With Stable Control Parameters of Planar Switched Reluctance Motors

How to Exhibit More Predictable Behaviors

Bridging State and History Representations: Understanding Self-Predictive RL

Unsupervised Representation Learning in Partially Observable Atari Games

State-wise Safe Reinforcement Learning: A Survey

An Overview of Natural Language State Representation for Reinforcement Learning

Model adaptive phase space reconstruction

State Sequences Prediction Via Fourier Transform for Representation Learning

Relationships between State Feedbacks of Completely and Partially Observable Discrete Event Systems

ProSpec RL: Plan Ahead, then Execute

Stochastic Observability and Convergent Analog State Estimation of Randomly Switched Linear Systems with Unobservable Subsystems