Abstract:Upper Confidence Bound (UCB) algorithms are a widely-used class of sequential algorithms for the $K$-armed bandit problem. Despite extensive research over the past decades aimed at understanding their asymptotic and (near) minimax optimality properties, a precise understanding of their regret behavior remains elusive. This gap has not only hindered the evaluation of their actual algorithmic efficiency, but also limited further developments in statistical inference in sequential data collection. This paper bridges these two fundamental aspects--precise regret analysis and adaptive statistical inference--through a deterministic characterization of the number of arm pulls for an UCB index algorithm [Lai87, Agr95, ACBF02]. Our resulting precise regret formula not only accurately captures the actual behavior of the UCB algorithm for finite time horizons and individual problem instances, but also provides significant new insights into the regimes in which the existing theory remains informative. In particular, we show that the classical Lai-Robbins regret formula is exact if and only if the sub-optimality gaps exceed the order $\sigma\sqrt{K\log T/T}$. We also show that its maximal regret deviates from the minimax regret by a logarithmic factor, and therefore settling its strict minimax optimality in the negative. The deterministic characterization of the number of arm pulls for the UCB algorithm also has major implications in adaptive statistical inference. Building on the seminal work of [Lai82], we show that the UCB algorithm satisfies certain stability properties that lead to quantitative central limit theorems in two settings including the empirical means of unknown rewards in the bandit setting. These results have an important practical implication: conventional confidence sets designed for i.i.d. data remain valid even when data are collected sequentially.

A central limit theorem, loss aversion and multi-armed bandits

Approximate optimality and the risk/reward tradeoff in a class of bandit problems

Multiarmed Bandits Problem Under the Mean-Variance Setting

Risk averse non-stationary multi-armed bandits

Optimal Algorithms for Lipschitz Bandits with Heavy-tailed Rewards

Swimming in curved space or The Baron and the cat

Multi-Armed Bandits with Abstention

A Survey of Risk-Aware Multi-Armed Bandits

Risk-averse Contextual Multi-armed Bandit Problem with Linear Payoffs

Combinatorial Multi-Armed Bandit: General Framework and Applications.

A Finite-Time Analysis of Multi-armed Bandits Problems with Kullback-Leibler Divergences

Understanding the stochastic dynamics of sequential decision-making processes: A path-integral analysis of multi-armed bandits

Zero-Inflated Bandits

A Unifying Theory of Thompson Sampling for Continuous Risk-Averse Bandits

Stochastic Graphical Bandits with Heavy-Tailed Rewards.

Combinatorial Bandits with Linear Constraints: Beyond Knapsacks and Fairness

On Lai's Upper Confidence Bound in Multi-Armed Bandits

Data-Driven Upper Confidence Bounds with Near-Optimal Regret for Heavy-Tailed Bandits

Non-stationary Bandits with Habituation and Recovery Dynamics and Knapsack Constraints

Finite Budget Analysis of Multi-Armed Bandit Problems.

UCB algorithms for multi-armed bandits: Precise regret and adaptive inference