Abstract:Human feedback is often used, either directly or indirectly, as input to algorithmic decision making. However, humans are biased: if the algorithm that takes as input the human feedback does not control for potential biases, this might result in biased algorithmic decision making, which can have a tangible impact on people's lives. In this paper, we study how to detect and correct for evaluators' bias in the task of ranking people (or items) from pairwise comparisons . Specifically, we assume we are given pairwise comparisons of the items to be ranked produced by a set of evaluators. While the pairwise assessments of the evaluators should reflect to a certain extent the latent (unobservable) true quality scores of the items, they might be affected by each evaluator's own bias against, or in favor, of some groups of items. By detecting and amending evaluators' biases, we aim to produce a ranking of the items that is, as much as possible, in accordance with the ranking one would produce by having access to the latent quality scores. Our proposal is a novel method that extends the classic Bradley-Terry model by having a bias parameter for each evaluator which distorts the true quality score of each item, depending on the group the item belongs to. Thanks to the simplicity of the model, we are able to write explicitly its log-likelihood w.r.t. the parameters (i.e., items' latent scores and evaluators' bias) and optimize by means of the alternating approach. Our experiments on synthetic and real-world data confirm that our method is able to reconstruct the bias of each single evaluator extremely well and thus to outperform several non-trivial competitors in the task of producing a ranking which is as much as possible close to the unbiased ranking.

Pairwise Learning for Personalized Ranking with Noisy Comparisons

Sebpr: Semantics Enhanced Bayesian Personalized Ranking With Comparable Item Pairs

Rankmbpr: Rank-Aware Mutual Bayesian Personalized Ranking For Item Recommendationl

Debiased Explainable Pairwise Ranking from Implicit Feedback

CPLR: Collaborative pairwise learning to rank for personalized recommendation.

Sampler Design for Bayesian Personalized Ranking by Leveraging View Data

Adaptive Bayesian Personalized Ranking for Heterogeneous Implicit Feedbacks

Using Entropy for Group Sampling in Pairwise Ranking from Implicit Feedback

Pareto Pairwise Ranking for Fairness Enhancement of Recommender Systems

Unbiased Pairwise Learning to Rank in Recommender Systems

Double bayesian pairwise learning for one-class collaborative filtering

M-BPR: A novel approach to improving BPR for recommendation with multi-type pair-wise preferences

Adversarial Personalized Ranking for Recommendation

Bayesian personalized ranking based on multiple-layer neighborhoods

Rate-Optimal Rank Aggregation with Private Pairwise Rankings

Adaptive Pairwise Preference Learning for Collaborative Recommendation with Implicit Feedbacks.

Bias-aware ranking from pairwise comparisons

Multiple Pairwise Ranking with Implicit Feedback.

Cross Pairwise Ranking for Unbiased Item Recommendation

Bi-Group Bayesian Personalized Ranking from Implicit Feedback

Prior-based bayesian pairwise ranking for one-class collaborative filtering