Abstract:The practicality of reinforcement learning algorithms has been limited due to poor scaling with respect to the problem size, as the sample complexity of learning an $\epsilon$-optimal policy is $\tilde{\Omega}\left(|S||A|H^3 / \epsilon^2\right)$ over worst case instances of an MDP with state space $S$, action space $A$, and horizon $H$. We consider a class of MDPs for which the associated optimal $Q^*$ function is low rank, where the latent features are unknown. While one would hope to achieve linear sample complexity in $|S|$ and $|A|$ due to the low rank structure, we show that without imposing further assumptions beyond low rank of $Q^*$, if one is constrained to estimate the $Q$ function using only observations from a subset of entries, there is a worst case instance in which one must incur a sample complexity exponential in the horizon $H$ to learn a near optimal policy. We subsequently show that under stronger low rank structural assumptions, given access to a generative model, Low Rank Monte Carlo Policy Iteration (LR-MCPI) and Low Rank Empirical Value Iteration (LR-EVI) achieve the desired sample complexity of $\tilde{O}\left((|S|+|A|)\mathrm{poly}(d,H)/\epsilon^2\right)$ for a rank $d$ setting, which is minimax optimal with respect to the scaling of $|S|, |A|$, and $\epsilon$. In contrast to literature on linear and low-rank MDPs, we do not require a known feature mapping, our algorithm is computationally simple, and our results hold for long time horizons. Our results provide insights on the minimal low-rank structural assumptions required on the MDP with respect to the transition kernel versus the optimal action-value function.

Uncertainty-Aware Low-Rank Q-Matrix Estimation for Deep Reinforcement Learning

Sample Efficient Reinforcement Learning via Low-Rank Matrix Estimation

Controlling Estimation Error in Reinforcement Learning via Reinforced Operation

Tensor and Matrix Low-Rank Value-Function Approximation in Reinforcement Learning

Fundamental Limits of Reinforcement Learning in Environment with Endogeneous and Exogeneous Uncertainty

Provably Efficient CVaR RL in Low-rank MDPs

Better Value Estimation in Q-learning-based Multi-Agent Reinforcement Learning

MQES: Max-Q Entropy Search for Efficient Exploration in Continuous Reinforcement Learning

Distributional reinforcement learning with epistemic and aleatoric uncertainty estimation

Uncertainty Quantification for Operators in Online Reinforcement Learning

Data Efficient Deep Reinforcement Learning with Action-Ranked Temporal Difference Learning

Eigensubspace of Temporal-Difference Dynamics and How It Improves Value Approximation in Reinforcement Learning

$\Sqrt{n}$-Regret for Learning in Markov Decision Processes with Function Approximation and Low Bellman Rank

Utilizing Maximum Mean Discrepancy Barycenter for Propagating the Uncertainty of Value Functions in Reinforcement Learning

Randomized Exploration for Reinforcement Learning with General Value Function Approximation

Overcoming the Long Horizon Barrier for Sample-Efficient Reinforcement Learning with Latent Low-Rank Structure

ResQ: A Residual Q Function-based Approach for Multi-Agent Reinforcement Learning Value Factorization

Uncertainty in Bayesian Reinforcement Learning for Robot Manipulation Tasks with Sparse Rewards.

Matrix Estimation for Offline Reinforcement Learning with Low-Rank Structure

Towards Safe Reinforcement Learning Via Constraining Conditional Value-at-Risk

Ensemble Network Architecture for Deep Reinforcement Learning