Abstract:Upper Confidence Bound (UCB) algorithms are a widely-used class of sequential algorithms for the $K$-armed bandit problem. Despite extensive research over the past decades aimed at understanding their asymptotic and (near) minimax optimality properties, a precise understanding of their regret behavior remains elusive. This gap has not only hindered the evaluation of their actual algorithmic efficiency, but also limited further developments in statistical inference in sequential data collection. This paper bridges these two fundamental aspects--precise regret analysis and adaptive statistical inference--through a deterministic characterization of the number of arm pulls for an UCB index algorithm [Lai87, Agr95, ACBF02]. Our resulting precise regret formula not only accurately captures the actual behavior of the UCB algorithm for finite time horizons and individual problem instances, but also provides significant new insights into the regimes in which the existing theory remains informative. In particular, we show that the classical Lai-Robbins regret formula is exact if and only if the sub-optimality gaps exceed the order $\sigma\sqrt{K\log T/T}$. We also show that its maximal regret deviates from the minimax regret by a logarithmic factor, and therefore settling its strict minimax optimality in the negative. The deterministic characterization of the number of arm pulls for the UCB algorithm also has major implications in adaptive statistical inference. Building on the seminal work of [Lai82], we show that the UCB algorithm satisfies certain stability properties that lead to quantitative central limit theorems in two settings including the empirical means of unknown rewards in the bandit setting. These results have an important practical implication: conventional confidence sets designed for i.i.d. data remain valid even when data are collected sequentially.

Tight Memory-Regret Lower Bounds for Streaming Bandits.

Tight Regret Bounds for Single-pass Streaming Multi-armed Bandits

A Sharp Memory-Regret Trade-Off for Multi-Pass Streaming Bandits

Streaming Algorithms for Stochastic Multi-armed Bandits

Optimal Streaming Algorithms for Multi-Armed Bandits

Tight Bounds for Bandit Combinatorial Optimization

Nearly Optimal Regret for Stochastic Linear Bandits with Heavy-Tailed Payoffs

Tight Regret Bounds for Infinite-armed Linear Contextual Bandits

Finite-Time Logarithmic Bayes Regret Upper Bounds

Chained Information-Theoretic bounds and Tight Regret Rate for Linear Bandit Problems

Batch-Size Independent Regret Bounds for Combinatorial Semi-Bandits with Probabilistically Triggered Arms or Independent Arms

Optimal Regret Is Achievable with Bounded Approximate Inference Error: An Enhanced Bayesian Upper Confidence Bound Framework

Tight Rates for Bandit Control Beyond Quadratics

Regret Bounds for Batched Bandits

Nash Regret Guarantees for Linear Bandits

Adaptive Regret for Bandits Made Possible: Two Queries Suffice

Non-stationary Linear Bandits Revisited

Lower Bounds for Time-Varying Kernelized Bandits

UCB algorithms for multi-armed bandits: Precise regret and adaptive inference

Equipping Experts/Bandits with Long-term Memory.

A General Recipe for the Analysis of Randomized Multi-Armed Bandit Algorithms