Abstract:Contextual bandits constitute a classical framework for decision-making under uncertainty. In this setting, the goal is to learn the arms of highest reward subject to contextual information, while the unknown reward parameters of each arm need to be learned by experimenting that specific arm. Accordingly, a fundamental problem is that of balancing exploration (i.e., pulling different arms to learn their parameters), versus exploitation (i.e., pulling the best arms to gain reward). To study this problem, the existing literature mostly considers perfectly observed contexts. However, the setting of partial context observations remains unexplored to date, despite being theoretically more general and practically more versatile. We study bandit policies for learning to select optimal arms based on the data of observations, which are noisy linear functions of the unobserved context vectors. Our theoretical analysis shows that the Thompson sampling policy successfully balances exploration and exploitation. Specifically, we establish the followings: (i) regret bounds that grow poly-logarithmically with time, (ii) square-root consistency of parameter estimation, and (iii) scaling of the regret with other quantities including dimensions and number of arms. Extensive numerical experiments with both real and synthetic data are presented as well, corroborating the efficacy of Thompson sampling. To establish the results, we introduce novel martingale techniques and concentration inequalities to address partially observed dependent random variables generated from unspecified distributions, and also leverage problem-dependent information to sharpen probabilistic bounds for time-varying suboptimality gaps. These techniques pave the road towards studying other decision-making problems with contextual information as well as partial observations.

Contextual Linear Bandits under Noisy Features: Towards Bayesian Oracles

Thompson Sampling for Stochastic Bandits with Noisy Contexts: An Information-Theoretic Regret Analysis

Stochastic Conservative Contextual Linear Bandits

Variance-Aware Linear UCB with Deep Representation for Neural Contextual Bandits

Thompson Sampling in Partially Observable Contextual Bandits

Stochastic Bandits with Context Distributions

Noise-Adaptive Confidence Sets for Linear Bandits and Application to Bayesian Optimization

Contexts can be Cheap: Solving Stochastic Contextual Bandits with Linear Bandit Algorithms

On the Optimal Regret of Locally Private Linear Contextual Bandit

Instance-Dependent Complexity of Contextual Bandits and Reinforcement Learning: A Disagreement-Based Perspective

Contextual Continuum Bandits: Static Versus Dynamic Regret

Robust Stochastic Linear Contextual Bandits Under Adversarial Attacks

Robust Contextual Linear Bandits.

Contextual Bandits with Stage-wise Constraints

Best-of-Both-Worlds Linear Contextual Bandits

Second Order Bounds for Contextual Bandits with Function Approximation

Provably Optimal Algorithms for Generalized Linear Contextual Bandits

Contextual Bandits for Unbounded Context Distributions

A Bayesian Approach for Subset Selection in Contextual Bandits.

Efficient and Robust High-Dimensional Linear Contextual Bandits.