Abstract:We propose a new sequential decision-making setting, combining key aspects of two established online learning problems with bandit feedback. The optimal action to play at any given moment is contingent on an underlying changing state which is not directly observable by the agent. Each state is associated with a context distribution, possibly corrupted, allowing the agent to identify the state. Furthermore, states evolve in a Markovian fashion, providing useful information to estimate the current state via state history. In the proposed problem setting, we tackle the challenge of deciding on which of the two sources of information the agent should base its arm selection. We present an algorithm that uses a referee to dynamically combine the policies of a contextual bandit and a multi-armed bandit. We capture the time-correlation of states through iteratively learning the action-reward transition model, allowing for efficient exploration of actions. Our setting is motivated by adaptive mobile health (mHealth) interventions. Users transition through different, time-correlated, but only partially observable internal states, determining their current needs. The side information associated with each internal state might not always be reliable, and standard approaches solely rely on the context risk of incurring high regret. Similarly, some users might exhibit weaker correlations between subsequent states, leading to approaches that solely rely on state transitions risking the same. We analyze our setting and algorithm in terms of regret lower bound and upper bounds and evaluate our method on simulated medication adherence intervention data and several real-world data sets, showing improved empirical performance compared to several popular algorithms.

Robust Contextual Bandit via the Capped- Norm for Mobile Health Intervention

Robust Contextual Bandit via the Capped-$$\ell _{2}$$ Norm for Mobile Health Intervention

Robust Contextual Bandit Via the Capped-ℓ_2 Norm

Robust Contextual Bandit via the Capped-` 2 norm

Robust Actor-Critic Contextual Bandit for Mobile Health (mHealth) Interventions

A Robust Mixed-Effects Bandit Algorithm for Assessing Mobile Health Interventions

Risk-Aware Continuous Control with Neural Contextual Bandits

A New Bandit Setting Balancing Information from State Evolution and Corrupted Context

Thompson sampling for zero-inflated count outcomes with an application to the Drink Less mobile health study

Partially Observable Contextual Bandits with Linear Payoffs

Contextual Bandits with Budgeted Information Reveal

Non-stationary Bandits with Habituation and Recovery Dynamics and Knapsack Constraints

A Bayesian Approach to Online Learning for Contextual Restless Bandits with Applications to Public Health

Stochastic Conservative Contextual Linear Bandits

Robust Tests in Online Decision-Making

Contextual Bandits with Random Projection

Contextual Bandits for Unbounded Context Distributions

Learning Context-Aware Probabilistic Maximum Coverage Bandits: A Variance-Adaptive Approach

Bayesian Collaborative Bandits with Thompson Sampling for Improved Outreach in Maternal Health Program

Neural Contextual Combinatorial Bandit under Non-stationary Environment

Proportional Response: Contextual Bandits for Simple and Cumulative Regret Minimization