Abstract:We propose a new sequential decision-making setting, combining key aspects of two established online learning problems with bandit feedback. The optimal action to play at any given moment is contingent on an underlying changing state which is not directly observable by the agent. Each state is associated with a context distribution, possibly corrupted, allowing the agent to identify the state. Furthermore, states evolve in a Markovian fashion, providing useful information to estimate the current state via state history. In the proposed problem setting, we tackle the challenge of deciding on which of the two sources of information the agent should base its arm selection. We present an algorithm that uses a referee to dynamically combine the policies of a contextual bandit and a multi-armed bandit. We capture the time-correlation of states through iteratively learning the action-reward transition model, allowing for efficient exploration of actions. Our setting is motivated by adaptive mobile health (mHealth) interventions. Users transition through different, time-correlated, but only partially observable internal states, determining their current needs. The side information associated with each internal state might not always be reliable, and standard approaches solely rely on the context risk of incurring high regret. Similarly, some users might exhibit weaker correlations between subsequent states, leading to approaches that solely rely on state transitions risking the same. We analyze our setting and algorithm in terms of regret lower bound and upper bounds and evaluate our method on simulated medication adherence intervention data and several real-world data sets, showing improved empirical performance compared to several popular algorithms.

Is Reinforcement Learning More Difficult Than Bandits? A Near-optimal Algorithm Escaping the Curse of Horizon.

Near-Optimal Regret Bounds for Multi-batch Reinforcement Learning

Is Long Horizon Reinforcement Learning More Difficult Than Short Horizon Reinforcement Learning?

Nearly Minimax Optimal Reward-free Reinforcement Learning.

Breaking the sample complexity barrier to regret-optimal model-free reinforcement learning

Provably Efficient Reinforcement Learning for Adversarial Restless Multi-Armed Bandits with Unknown Transitions and Bandit Feedback

Is Long Horizon Rl More Difficult Than Short Horizon RL?

A New Bandit Setting Balancing Information from State Evolution and Corrupted Context

Horizon-Free Reinforcement Learning in Polynomial Time: the Power of Stationary Policies

Context-lumpable stochastic bandits

Horizon-Free and Variance-Dependent Reinforcement Learning for Latent Markov Decision Processes

Near-Optimal Reinforcement Learning with Self-Play under Adaptivity Constraints

Posterior Sampling-based Online Learning for Episodic POMDPs

Online Reinforcement Learning in Markov Decision Process Using Linear Programming

Bypassing the Monster: A Faster and Simpler Optimal Algorithm for Contextual Bandits under Realizability

Nearly Horizon-Free Offline Reinforcement Learning

Provably Efficient Reinforcement Learning Via Surprise Bound

Learning Adversarial MDPs with Bandit Feedback and Unknown Transition

Overcoming the Long Horizon Barrier for Sample-Efficient Reinforcement Learning with Latent Low-Rank Structure

Almost Optimal Model-Free Reinforcement Learning Via Reference-Advantage Decomposition