Abstract:In digital health and EdTech, recommendation systems face a significant challenge: users often choose impulsively, in ways that conflict with the platform's long-term payoffs. This misalignment makes it difficult to effectively learn to rank items, as it may hinder exploration of items with greater long-term payoffs. Our paper tackles this issue by utilizing users' limited attention spans. We propose a model where a platform presents items with unknown payoffs to the platform in a ranked list to $T$ users over time. Each user selects an item by first considering a prefix window of these ranked items and then picking the highest preferred item in that window (and the platform observes its payoff for this item). We study the design of online bandit algorithms that obtain vanishing regret against hindsight optimal benchmarks. We first consider adversarial window sizes and stochastic iid payoffs. We design an active-elimination-based algorithm that achieves an optimal instance-dependent regret bound of $O(\log(T))$, by showing matching regret upper and lower bounds. The key idea is using the combinatorial structure of the problem to either obtain a large payoff from each item or to explore by getting a sample from that item. This method systematically narrows down the item choices to enhance learning efficiency and payoff. Second, we consider adversarial payoffs and stochastic iid window sizes. We start from the full-information problem of finding the permutation that maximizes the expected payoff. By a novel combinatorial argument, we characterize the polytope of admissible item selection probabilities by a permutation and show it has a polynomial-size representation. Using this representation, we show how standard algorithms for adversarial online linear optimization in the space of admissible probabilities can be used to obtain a polynomial-time algorithm with $O(\sqrt{T})$ regret.

Adversarial Attacks on Online Learning to Rank with Click Feedback

Adversarial Attacks on Online Learning to Rank with Stochastic Click Models

Simultaneously Learning Stochastic and Adversarial Bandits under the Position-Based Model

Adversarial Attacks on Adversarial Bandits

Exploration of Unranked Items in Safe Online Learning to Re-Rank

Reinforcement Online Learning to Rank with Unbiased Reward Shaping

LT2R: Learning to Online Learning to Rank for Web Search

Non-stationary Dueling Bandits for Online Learning to Rank

TopRank: A Practical Algorithm for Online Stochastic Ranking

Bandit Learning to Rank with Position-Based Click Models: Personalized and Equal Treatments

Effective and secure federated online learning to rank

Online Learning to Rank with List-level Feedback for Image Filtering

Understanding Adversarial Attacks on Observations in Deep Reinforcement Learning

Dynamics of Adversarial Attacks on Large Language Model-Based Search Engines

Misalignment, Learning, and Ranking: Harnessing Users Limited Attention

Optimal Attack and Defense for Reinforcement Learning

Adversarial Online Learning with noise

Adversaries in Online Learning Revisited: with applications in Robust Optimization and Adversarial training

Scalable Exploration for Neural Online Learning to Rank with Perturbed Feedback

Stochastic Bandits Robust to Adversarial Attacks

Practical Order Attack in Deep Ranking