Abstract:We study a $K$-armed non-stationary bandit model where rewards change smoothly, as captured by Hölder class assumptions on rewards as functions of time. Such smooth changes are parametrized by a Hölder exponent $\beta$ and coefficient $\lambda$. While various sub-cases of this general model have been studied in isolation, we first establish the minimax dynamic regret rate generally for all $K,\beta,\lambda$. Next, we show this optimal dynamic regret can be attained adaptively, without knowledge of $\beta,\lambda$. To contrast, even with parameter knowledge, upper bounds were only previously known for limited regimes $\beta\leq 1$ and $\beta=2$ (Slivkins, 2014; Krishnamurthy and Gopalan, 2021; Manegueu et al., 2021; Jia et al.,2023). Thus, our work resolves open questions raised by these disparate threads of the literature. We also study the problem of attaining faster gap-dependent regret rates in non-stationary bandits. While such rates are long known to be impossible in general (Garivier and Moulines, 2011), we show that environments admitting a safe arm (Suk and Kpotufe, 2022) allow for much faster rates than the worst-case scaling with $\sqrt{T}$. While previous works in this direction focused on attaining the usual logarithmic regret bounds, as summed over stationary periods, our new gap-dependent rates reveal new optimistic regimes of non-stationarity where even the logarithmic bounds are pessimistic. We show our new gap-dependent rate is tight and that its achievability (i.e., as made possible by a safe arm) has a surprisingly simple and clean characterization within the smooth Hölder class model.

A Simple Approach For Non-Stationary Linear Bandits

Non-stationary Linear Bandits Revisited

Revisiting Weighted Strategy for Non-stationary Parametric Bandits

Nearly Optimal Regret for Stochastic Linear Bandits with Heavy-Tailed Payoffs

Combinatorial Semi-Bandit in the Non-Stationary Environment

Adaptive Smooth Non-Stationary Bandits

Non-stationary Bandits with Habituation and Recovery Dynamics and Knapsack Constraints

A Risk-Averse Framework for Non-Stationary Stochastic Multi-Armed Bandits

Variance-Dependent Regret Bounds for Non-stationary Linear Bandits

Restless Linear Bandits

Multiscale Non-stationary Stochastic Bandits

Near-Optimal Algorithm for Non-Stationary Kernelized Bandits

Non-Stationary Latent Auto-Regressive Bandits

Diminishing Exploration: A Minimalist Approach to Piecewise Stationary Multi-Armed Bandits

Bandit Convex Optimization in Non-stationary Environments.

Nearly Minimax-Optimal Regret for Linearly Parameterized Bandits.

Stochastic Conservative Contextual Linear Bandits

Stochastic Bandits with Graph Feedback in Non-Stationary Environments

Learning Contextual Bandits in a Non-stationary Environment

Efficient Change-Point Detection for Tackling Piecewise-Stationary Bandits

Tight Rates for Bandit Control Beyond Quadratics