Abstract:We study reinforcement learning (RL) with linear function approximation where the underlying transition probability kernel of the Markov decision process (MDP) is a linear mixture model (Jia et al., 2020; Ayoub et al., 2020; Zhou et al., 2020) and the learning agent has access to either an integration or a sampling oracle of the individual basis kernels. We propose a new Bernstein-type concentration inequality for self-normalized martingales for linear bandit problems with bounded noise. Based on the new inequality, we propose a new, computationally efficient algorithm with linear function approximation named $\text{UCRL-VTR}^{+}$ for the aforementioned linear mixture MDPs in the episodic undiscounted setting. We show that $\text{UCRL-VTR}^{+}$ attains an $\tilde O(dH\sqrt{T})$ regret where $d$ is the dimension of feature mapping, $H$ is the length of the episode and $T$ is the number of interactions with the MDP. We also prove a matching lower bound $\Omega(dH\sqrt{T})$ for this setting, which shows that $\text{UCRL-VTR}^{+}$ is minimax optimal up to logarithmic factors. In addition, we propose the $\text{UCLK}^{+}$ algorithm for the same family of MDPs under discounting and show that it attains an $\tilde O(d\sqrt{T}/(1-\gamma)^{1.5})$ regret, where $\gamma\in [0,1)$ is the discount factor. Our upper bound matches the lower bound $\Omega(d\sqrt{T}/(1-\gamma)^{1.5})$ proved by Zhou et al. (2020) up to logarithmic factors, suggesting that $\text{UCLK}^{+}$ is nearly minimax optimal. To the best of our knowledge, these are the first computationally efficient, nearly minimax optimal algorithms for RL with linear function approximation.

Learning Infinite-Horizon Average-Reward Linear Mixture MDPs of Bounded Span

Provably Efficient Infinite-Horizon Average-Reward Reinforcement Learning with Linear Function Approximation

Reinforcement Learning for Infinite-Horizon Average-Reward Linear MDPs via Approximation by Discounted-Reward MDPs

Provably Efficient Reinforcement Learning for Infinite-Horizon Average-Reward Linear MDPs

Nearly Minimax Optimal Reinforcement Learning for Linear Mixture Markov Decision Processes

Dynamic Regret of Adversarial Linear Mixture MDPs

Infinite-Horizon Reinforcement Learning with Multinomial Logistic Function Approximation

Optimal Horizon-Free Reward-Free Exploration for Linear Mixture MDPs

Value-Biased Maximum Likelihood Estimation for Model-based Reinforcement Learning in Discounted Linear MDPs

Near-Optimal Dynamic Regret for Adversarial Linear Mixture MDPs

Achieving Tractable Minimax Optimal Regret in Average Reward MDPs

Nearly Minimax Optimal Regret for Learning Linear Mixture Stochastic Shortest Path

Improved Algorithm for Adversarial Linear Mixture MDPs with Bandit Feedback and Unknown Transition

Sample-efficient Learning of Infinite-horizon Average-reward MDPs with General Function Approximation

Nearly Minimax Optimal Reinforcement Learning with Linear Function Approximation.

Optimal Sample Complexity for Average Reward Markov Decision Processes

Span-Based Optimal Sample Complexity for Average Reward MDPs

Horizon-Free and Variance-Dependent Reinforcement Learning for Latent Markov Decision Processes

Sharper Model-free Reinforcement Learning for Average-reward Markov Decision Processes

Logarithmic Regret Bounds for Continuous-Time Average-Reward Markov Decision Processes