Abstract:User interaction data is an important source of supervision in counterfactual learning to rank (CLTR). Such data suffers from presentation bias. Much work in unbiased learning to rank (ULTR) focuses on position bias, i.e., items at higher ranks are more likely to be examined and clicked. Inter-item dependencies also influence examination probabilities, with outlier items in a ranking as an important example. Outliers are defined as items that observably deviate from the rest and therefore stand out in the ranking. In this paper, we identify and introduce the bias brought about by outlier items: users tend to click more on outlier items and their close neighbors. To this end, we first conduct a controlled experiment to study the effect of outliers on user clicks. Next, to examine whether the findings from our controlled experiment generalize to naturalistic situations, we explore real-world click logs from an e-commerce platform. We show that, in both scenarios, users tend to click significantly more on outlier items than on non-outlier items in the same rankings. We show that this tendency holds for all positions, i.e., for any specific position, an item receives more interactions when presented as an outlier as opposed to a non-outlier item. We conclude from our analysis that the effect of outliers on clicks is a type of bias that should be addressed in ULTR. We therefore propose an outlier-aware click model that accounts for both outlier and position bias, called outlier-aware position-based model ( OPBM). We estimate click propensities based on OPBM ; through extensive experiments performed on both real-world e-commerce data and semi-synthetic data, we verify the effectiveness of our outlier-aware click model. Our results show the superiority of OPBM against baselines in terms of ranking performance and true relevance estimation.

Scalar is Not Enough

Scalar is Not Enough: Vectorization-based Unbiased Learning to Rank

LBD: Decouple Relevance and Observation for Individual-Level Unbiased Learning to Rank

Model-based Unbiased Learning to Rank

Towards Disentangling Relevance and Bias in Unbiased Learning to Rank

Analysis of Multivariate Scoring Functions for Automatic Unbiased Learning to Rank

Click-Through Rate Prediction Algorithm Based on Modeling of Implicit High-Order Feature Importance

Unbiased Learning to Rank

Whole Page Unbiased Learning to Rank

Unbiased Learning-to-Rank with Biased Feedback

ULTRE framework: a framework for Unbiased Learning to Rank Evaluation based on simulation of user behavior

ULTRA: An Unbiased Learning To Rank Algorithm Toolbox

Identifiability Matters: Revealing the Hidden Recoverable Condition in Unbiased Learning to Rank

Unbiased Learning to Rank Meets Reality: Lessons from Baidu's Large-Scale Search Dataset

On the Impact of Outlier Bias on User Clicks

Unbiased Learning to Rank: Online or Offline?

Can Clicks Be Both Labels and Features?

Unbiased Learning-to-Rank Needs Unconfounded Propensity Estimation

Understanding the Effects of the Baidu-ULTR Logging Policy on Two-Tower Models

Contextual Dual Learning Algorithm with Listwise Distillation for Unbiased Learning to Rank

A Deep Recurrent Survival Model for Unbiased Ranking