Abstract:We study a $K$-armed non-stationary bandit model where rewards change smoothly, as captured by Hölder class assumptions on rewards as functions of time. Such smooth changes are parametrized by a Hölder exponent $\beta$ and coefficient $\lambda$. While various sub-cases of this general model have been studied in isolation, we first establish the minimax dynamic regret rate generally for all $K,\beta,\lambda$. Next, we show this optimal dynamic regret can be attained adaptively, without knowledge of $\beta,\lambda$. To contrast, even with parameter knowledge, upper bounds were only previously known for limited regimes $\beta\leq 1$ and $\beta=2$ (Slivkins, 2014; Krishnamurthy and Gopalan, 2021; Manegueu et al., 2021; Jia et al.,2023). Thus, our work resolves open questions raised by these disparate threads of the literature. We also study the problem of attaining faster gap-dependent regret rates in non-stationary bandits. While such rates are long known to be impossible in general (Garivier and Moulines, 2011), we show that environments admitting a safe arm (Suk and Kpotufe, 2022) allow for much faster rates than the worst-case scaling with $\sqrt{T}$. While previous works in this direction focused on attaining the usual logarithmic regret bounds, as summed over stationary periods, our new gap-dependent rates reveal new optimistic regimes of non-stationarity where even the logarithmic bounds are pessimistic. We show our new gap-dependent rate is tight and that its achievability (i.e., as made possible by a safe arm) has a surprisingly simple and clean characterization within the smooth Hölder class model.

Combinatorial Semi-Bandit in the Non-Stationary Environment

Beating Stochastic and Adversarial Semi-bandits Optimally and Simultaneously

Adversarial Combinatorial Bandits with Switching Costs

Bandit Convex Optimization in Non-stationary Environments.

Non-stationary Linear Bandits Revisited

A Simple Approach For Non-Stationary Linear Bandits

Piecewise-Stationary Combinatorial Semi-Bandit with Causally Related Rewards

Bridging Adversarial and Nonstationary Multi-armed Bandit

Near-Optimal Algorithm for Non-Stationary Kernelized Bandits

Variance-Dependent Regret Bounds for Non-stationary Linear Bandits

Adaptive Regret for Bandits Made Possible: Two Queries Suffice

Corralling a Larger Band of Bandits: A Case Study on Switching Regret for Linear Bandits

Simple Combinatorial Algorithms for Combinatorial Bandits - Corruptions and Approximations.

Combinatorial Stochastic-Greedy Bandit

Adaptive Smooth Non-Stationary Bandits

Stochastic Online Greedy Learning with Semi-Bandit Feedbacks

Non-Stationary Latent Auto-Regressive Bandits

Probably Anytime-Safe Stochastic Combinatorial Semi-Bandits

Non-stationary Bandits with Habituation and Recovery Dynamics and Knapsack Constraints

Combinatorial Multi-Armed Bandit with General Reward Functions

MNL-Bandit in non-stationary environments