Abstract:This paper presents new variance-aware confidence sets for linear bandits and linear mixture Markov Decision Processes (MDPs). With the new confidence sets, we obtain the follow regret bounds: For linear bandits, we obtain an (O) over tilde (poly(d)root 1 Sigma(K)(k=1) sigma(2)(k)) data-dependent regret bound, where d is the feature dimension, K is the number of rounds, and sigma(2)(k) is the unknown variance of the reward at the k-th round. This is the first regret bound that only scales with the variance and the dimension but no explicit polynomial dependency on K. When variances are small, this bound can be significantly smaller than the (Theta) over tilde (d root K) worst-case regret bound. For linear mixture MDPs, we obtain an (O) over tilde (poly(d,log H root K) regret bound, where d is the number of base models, K is the number of episodes, and H is the planning horizon. This is the first regret bound that only scales logarithmically with H in the reinforcement learning with linear function approximation setting, thus exponentially improving existing results, and resolving an open problem in [Zhou et al., 2020a]. We develop three technical ideas that may be of independent interest: 1) applications of the peeling technique to both the input norm and the variance magnitude, 2) a recursion-based estimator for the variance, and 3) a new convex potential lemma that generalizes the seminal elliptical potential lemma.

Information-Theoretic Confidence Bounds for Reinforcement Learning

Information-Theoretic Minimax Regret Bounds for Reinforcement Learning based on Duality

An Information-Theoretic Analysis of Bayesian Reinforcement Learning

Upper Counterfactual Confidence Bounds: a New Optimism Principle for Contextual Bandits

Fundamental Limits of Reinforcement Learning in Environment with Endogeneous and Exogeneous Uncertainty

An Information-Theoretic Optimality Principle for Deep Reinforcement Learning

Improved Algorithms for Stochastic Linear Bandits Using Tail Bounds for Martingale Mixtures

Chained Information-Theoretic bounds and Tight Regret Rate for Linear Bandit Problems

Optimistic Posterior Sampling for Reinforcement Learning: Worst-Case Regret Bounds

Information-Theoretic Generalization Bounds for Batch Reinforcement Learning

Optimistic Thompson Sampling for No-Regret Learning in Unknown Games

Optimistic Information Directed Sampling

Information Theoretic Lower Bounds for Information Theoretic Upper Bounds

Learning to Optimize via Posterior Sampling

An Information-Theoretic Analysis of Nonstationary Bandit Learning

Improved Variance-Aware Confidence Sets for Linear Bandits and Linear Mixture MDP

Information Entropy of Uncertainty Control: an Uncertainty Management Method in Imperfect Information Games

Variance-Aware Confidence Set: Variance-Dependent Bound for Linear Bandits and Horizon-Free Bound for Linear Mixture MDP

Self-supervised Sequential Information Bottleneck for Robust Exploration in Deep Reinforcement Learning

An Information Theoretic Approach to Interaction-Grounded Learning

Information Directed Sampling for Sparse Linear Bandits