Abstract:Learning to rank from logged user feedback, such as clicks or purchases, is a central component of many real-world information systems. Different from human-annotated relevance labels, the user feedback is always noisy and biased. Many existing learning to rank methods infer the underlying relevance of query–item pairs based on different assumptions of examination, and still optimize a relevance based objective. Such methods rely heavily on the correct estimation of examination, which is often difficult to achieve in practice. In this work, we propose a general framework U-rank+ for learning to rank with logged user feedback from the perspective of graph matching. We systematically analyze the biases in user feedback, including examination bias and selection bias. Then, we take both biases into consideration for unbiased utility estimation that directly based on user feedback, instead of relevance. In order to maximize the estimated utility in an efficient manner, we design two different solvers based on Sinkhorn and LambdaLoss for U-rank+ . The former is based on a standard graph matching algorithm, and the latter is inspired by the traditional method of learning to rank. Both of the algorithms have good theoretical properties to optimize the unbiased utility objective while the latter is proved to be empirically more effective and efficient in practice. Our framework U-rank+ can deal with a general utility function and can be used in a widespread of applications including web search, recommendation, and online advertising. Semi-synthetic experiments on three benchmark learning to rank datasets demonstrate the effectiveness of U-rank+ . Furthermore, our proposed framework has been deployed on two different scenarios of a mainstream App store, where the online A/B testing shows that U-rank+ achieves an average improvement of 19.2% on click-through rate and 20.8% improvement on conversion rate in recommendation scenario, and 5.12% on platform revenue in online advertising scenario over the production baselines.

Relevance and Irrelevance Graph Based Marginal Fisher Analysis for Image Search Reranking

Contextual Modeling on Auxiliary Points for Robust Image Reranking

Online Metric Learning for Relevance Feedback in E-Commerce Image Retrieval Applications

Contribution-Based Multi-Stream Feature Distance Fusion Method With <inline-formula> <tex-math notation="LaTeX">${k}$ </tex-math></inline-formula>-Distribution Re-Ranking for Person Re-Identification

Search Result Reranking with Visual and Structure Information Sources

Multi-Query Parallel Field Ranking for image retrieval.

Relevance Estimation with Multiple Information Sources on Search Engine Result Pages.

Image Retrieval Based on Fuzzy Semantic Relevance Matrix

Discriminative Multi-View Interactive Image Re-Ranking.

Visual Reranking through Weakly Supervised Multi-graph Learning

Exploiting Click Constraints and Multi-view Features for Image Re-ranking

DeepRank: A New Deep Architecture for Relevance Ranking in Information Retrieval

Maximizing Marginal Fairness for Dynamic Learning to Rank

Beyond Relevance Ranking: A General Graph Matching Framework for Utility-Oriented Learning to Rank

Efficient Manifold Ranking for Image Retrieval

Image Search Reranking with Query-Dependent Click-Based Relevance Feedback.

Weakly Supervised Multi-Graph Learning for Robust Image Reranking

Re-ranking image-text matching by adaptive metric fusion

A New Re-Ranking Method Using Enhanced Pseudo-Relevance Feedback for Content-Based Medical Image Retrieval

Learning to Rank Using User Clicks and Visual Features for Image Retrieval

Visual reranking with improved image graph