Abstract:Confronted with the challenge of identifying the most suitable metric to validate the merits of newly proposed models, the decision-making process is anything but straightforward. Given that comparing rankings introduces its own set of formidable challenges and the likely absence of a universal metric applicable to all scenarios, the scenario does not get any better. Furthermore, metrics designed for specific contexts, such as for Recommender Systems, sometimes extend to other domains without a comprehensive grasp of their underlying mechanisms, resulting in unforeseen outcomes and potential misuses. Complicating matters further, distinct metrics may emphasize different aspects of rankings, frequently leading to seemingly contradictory comparisons of model results and hindering the trustworthiness of evaluations. We unveil these aspects in the domain of ranking evaluation metrics. Firstly, we show instances resulting in inconsistent evaluations, sources of potential mistrust in commonly used metrics; by quantifying the frequency of such disagreements, we prove that these are common in rankings. Afterward, we conceptualize rankings using the mathematical formalism of symmetric groups detaching from possible domains where the metrics have been created; through this approach, we can rigorously and formally establish essential mathematical properties for ranking evaluation metrics, essential for a deeper comprehension of the source of inconsistent evaluations. We conclude with a discussion, connecting our theoretical analysis to the practical applications, highlighting which properties are important in each domain where rankings are commonly evaluated. In conclusion, our analysis sheds light on ranking evaluation metrics, highlighting that inconsistent evaluations should not be seen as a source of mistrust but as the need to carefully choose how to evaluate our models in the future.

Ranking evaluation metrics from a group-theoretic perspective

Metric-agnostic Ranking Optimization

A Closer Look at Classification Evaluation Metrics and a Critical Reflection of Common Evaluation Practice

Batch Evaluation Metrics in Information Retrieval: Measures, Scales, and Meaning

On Obtaining Stable Rankings

A novel evaluation methodology for supervised Feature Ranking algorithms

Never mind the metrics -- what about the uncertainty? Visualising confusion matrix metric distributions

Rankers, Rankees, & Rankings: Peeking into the Pandora's Box from a Socio-Technical Perspective

An Axiomatic Analysis of Diversity Evaluation Metrics: Introducing the Rank-Biased Utility Metric

Fair ranking: a critical review, challenges, and future directions

Rank-Preference Consistency as the Appropriate Metric for Recommender Systems

Models Versus Satisfaction: Towards a Better Understanding of Evaluation Metrics

Foundations of the Theory of Performance-Based Ranking

Robust Consensus in Ranking Data Analysis: Definitions, Properties and Computational Issues

Inconsistency of evaluation metrics in link prediction

Unbiased Comparative Evaluation of Ranking Functions

Fairness in Recommendation Ranking through Pairwise Comparisons

Ranking Multi-Metric Scientific Achievements Using a Concept of Pareto Optimality

Fairness in Ranking: A Survey

Ranking with Ties based on Noisy Performance Data

Quantifying the Impact of User Attention on Fair Group Representation in Ranked Lists