Abstract:Contextual bandit algorithms are sensitive to the estimation method of the outcome model as well as the exploration method used, particularly in the presence of rich heterogeneity or complex outcome models, which can lead to difficult estimation problems along the path of learning. We study a consideration for the exploration vs. exploitation framework that does not arise in multi-armed bandits but is crucial in contextual bandits; the way exploration and exploitation is conducted in the present affects the bias and variance in the potential outcome model estimation in subsequent stages of learning. We develop parametric and non-parametric contextual bandits that integrate balancing methods from the causal inference literature in their estimation to make it less prone to problems of estimation bias. We provide the first regret bound analyses for contextual bandits with balancing in the domain of linear contextual bandits that match the state of the art regret bounds. We demonstrate the strong practical advantage of balanced contextual bandits on a large number of supervised learning datasets and on a synthetic example that simulates model mis-specification and prejudice in the initial training data. Additionally, we develop contextual bandits with simpler assignment policies by leveraging sparse model estimation methods from the econometrics literature and demonstrate empirically that in the early stages they can improve the rate of learning and decrease regret.

Incentivising Exploration and Recommendations for Contextual Bandits with Payments

Partially Observable Contextual Bandits with Linear Payoffs

Strategic Linear Contextual Bandits

Designing an Interpretable Interface for Contextual Bandits

Selectively Contextual Bandits

Learning Contextual Bandits in a Non-stationary Environment

Bayesian Incentive-Compatible Bandit Exploration

Incentivizing Exploration with Linear Contexts and Combinatorial Actions

Estimation Considerations in Contextual Bandits

Incentivizing Exploration in Linear Bandits under Information Gap

Contextual Bandits with Budgeted Information Reveal

Counterfactual contextual bandit for recommendation under delayed feedback

Follow-ups Also Matter: Improving Contextual Bandits via Post-serving Contexts

Contextual Bandits with Random Projection

Incentivized Exploration of Non-Stationary Stochastic Bandits

Latent Contextual Bandits and their Application to Personalized Recommendations for New Users

Online learning with Corrupted context: Corrupted Contextual Bandits

Batched Online Contextual Sparse Bandits with Sequential Inclusion of Features

Privacy-Preserving Multi-Party Contextual Bandits

Conditionally Risk-Averse Contextual Bandits

Causal Contextual Bandits with Adaptive Context