Abstract:This paper provides a unified view of loss functions used in learning to rank. Loss function is a key component in learning to rank, because it encodes human knowledge on evaluation of ranking and guides the process of learning. Many loss functions have been proposed in the literature of learning to rank, with different forms and different motivations, and have been exploited in the development of various algorithms. However, there are several questions not well answered yet regarding the loss functions: (i) what is the relationship among them? (ii) can the optimization of them lead to the maximization of IR measures such as NDCG? (iii) how can we modify them to improve the performance of learning algorithms? In this paper, we try to answer these questions by proposing a quantity named ‘the unified loss for ranking’. The unified loss is defined by modeling ranking as a sequence of classification tasks, and assigning different weights to the tasks to reflect their importance in the overall ranking problem. With regards to the unified loss, we obtain the following three theoretical results. First, the loss functions in most existing algorithms, such as Ranking SVM, RankNet, RankBoost, and ListMLE, are margin-based surrogate losses (and also upper bounds) of the unified loss with uniform weight. Second, the unified loss is an upper bound of one minus NDCG, when the weights are set according to the position discount function and the gain function in NDCG. Third, if we can modify the loss functions of existing algorithms by introducing the weights in the unified loss, they are still upper bounds of the corresponding unified loss, and thus upper bounds of one minus NDCG. According to the results, we can expect that the minimization of the modified loss functions will lead to the effective maximization of NDCG. We have performed experiments on benchmark datasets to validate our theoretical findings. The results show that the ranking performances of the models trained with the modified loss functions can be much better than those of the models trained with the original loss functions.

RANK-META: A two-stage ranking algorithm based on meta-learning

A new ranking model constructing algorithm based on direct optimization of NDCG

Construction Algorithm of ranking Model Based on Non-Convex Upper Bound

A Unified Meta-learning Framework for Fair Ranking with Curriculum Learning

Ranking Measures and Loss Functions in Learning to Rank.

A Clustering-Ranking Method for Many-Objective Optimization

Large-scale Stochastic Optimization of NDCG Surrogates for Deep Learning with Provable Convergence

Directly optimizing evaluation measures in learning to rank.

AdaRank: a boosting algorithm for information retrieval.

Adaptive Neural Ranking Framework: Toward Maximized Business Goal for Cascade Ranking Systems

Meta Learning to Rank for Sparsely Supervised Queries

GreedyAgent: Crafting Efficient Agents for Meta-learning from Learning Curves Via Greedy Algorithm Selection

Optimizing Preference Alignment with Differentiable NDCG Ranking

A Unified View of Loss Functions in Learning to Rank

Ranking with Deep Multi-Objective Learning

A Theoretical Analysis of NDCG Type Ranking Measures.

Rank learning on training set selection and image quality assessment

Top-N-Rank: A Scalable List-wise Ranking Method for Recommender Systems

Learning to Re-rank with Constrained Meta-Optimal Transport

Offline Model-Based Optimization by Learning to Rank

AutoML for Large Capacity Modeling of Meta's Ranking Systems