Abstract:We propose a new sequential decision-making setting, combining key aspects of two established online learning problems with bandit feedback. The optimal action to play at any given moment is contingent on an underlying changing state which is not directly observable by the agent. Each state is associated with a context distribution, possibly corrupted, allowing the agent to identify the state. Furthermore, states evolve in a Markovian fashion, providing useful information to estimate the current state via state history. In the proposed problem setting, we tackle the challenge of deciding on which of the two sources of information the agent should base its arm selection. We present an algorithm that uses a referee to dynamically combine the policies of a contextual bandit and a multi-armed bandit. We capture the time-correlation of states through iteratively learning the action-reward transition model, allowing for efficient exploration of actions. Our setting is motivated by adaptive mobile health (mHealth) interventions. Users transition through different, time-correlated, but only partially observable internal states, determining their current needs. The side information associated with each internal state might not always be reliable, and standard approaches solely rely on the context risk of incurring high regret. Similarly, some users might exhibit weaker correlations between subsequent states, leading to approaches that solely rely on state transitions risking the same. We analyze our setting and algorithm in terms of regret lower bound and upper bounds and evaluate our method on simulated medication adherence intervention data and several real-world data sets, showing improved empirical performance compared to several popular algorithms.

Self-fulfilling Bandits: Endogeneity Spillover and Dynamic Selection in Algorithmic Decision-making

Dynamic Selection in Algorithmic Decision-making

Robustly Improving Bandit Algorithms with Confounded and Selection Biased Offline Data: A Causal Approach

Statistical Inference for Online Decision-Making: In a Contextual Bandit Setting

Regret Minimization and Statistical Inference in Online Decision Making with High-dimensional Covariates

Biased Dueling Bandits with Stochastic Delayed Feedback

A New Bandit Setting Balancing Information from State Evolution and Corrupted Context

Dynamic Batch Learning in High-Dimensional Sparse Linear Contextual Bandits

Strategic Linear Contextual Bandits

Multi-armed linear bandits with latent biases

Adaptive Algorithm for Multi-Armed Bandit Problem with High-Dimensional Covariates

Estimation Considerations in Contextual Bandits

Principal-Agent Bandit Games with Self-Interested and Exploratory Learning Agents

Bayesian Incentive-Compatible Bandit Exploration

Asymptotic Theory for IV-Based Reinforcement Learning with Potential Endogeneity

Dimension Reduction in Contextual Online Learning Via Nonparametric Variable Selection

Partially Observable Contextual Bandits with Linear Payoffs

Networked Bandits With Disjoint Linear Payoffs

Decentralized Nash Equilibria Learning for Online Game with Bandit Feedback

Distributed Online Bandit Linear Regressions with Differential Privacy.

Learning Contextual Bandits in a Non-stationary Environment