Abstract:Confronted with the challenge of identifying the most suitable metric to validate the merits of newly proposed models, the decision-making process is anything but straightforward. Given that comparing rankings introduces its own set of formidable challenges and the likely absence of a universal metric applicable to all scenarios, the scenario does not get any better. Furthermore, metrics designed for specific contexts, such as for Recommender Systems, sometimes extend to other domains without a comprehensive grasp of their underlying mechanisms, resulting in unforeseen outcomes and potential misuses. Complicating matters further, distinct metrics may emphasize different aspects of rankings, frequently leading to seemingly contradictory comparisons of model results and hindering the trustworthiness of evaluations. We unveil these aspects in the domain of ranking evaluation metrics. Firstly, we show instances resulting in inconsistent evaluations, sources of potential mistrust in commonly used metrics; by quantifying the frequency of such disagreements, we prove that these are common in rankings. Afterward, we conceptualize rankings using the mathematical formalism of symmetric groups detaching from possible domains where the metrics have been created; through this approach, we can rigorously and formally establish essential mathematical properties for ranking evaluation metrics, essential for a deeper comprehension of the source of inconsistent evaluations. We conclude with a discussion, connecting our theoretical analysis to the practical applications, highlighting which properties are important in each domain where rankings are commonly evaluated. In conclusion, our analysis sheds light on ranking evaluation metrics, highlighting that inconsistent evaluations should not be seen as a source of mistrust but as the need to carefully choose how to evaluate our models in the future.

A New Weighted Spearman's Footrule as A Measure of Distance between Rankings

Accuracy Estimation of Link-Based Similarity Measures and Its Application

Weighting by Tying: A New Approach to Weighted Rank Correlation

Approximate Rank Aggregation

A New Correlation Coefficient for Aggregating Non-strict and Incomplete Rankings

Graph-based Rank Aggregation Method for High-Dimensional and Partial Rankings

Fair Rank Aggregation

Multivariate Spearman's rho for aggregating ranks using copulas

On the Weighted Top-Difference Distance: Axioms, Aggregation, and Approximation

Relative Euclidean Distance with Application to TOPSIS and Estimation Performance Ranking

Approximate Ranking from Pairwise Comparisons

Measures for Ranking Estimation Performance Based on Single or Multiple Performance Metrics

Approximate and Dynamic Rank Aggregation

Disparity-Based Measurement on Participation Interest and Competition Recommendation Method

Ranking evaluation metrics from a group-theoretic perspective

A Two-Stage Ranking Method To Minimize Ordinal Violation For Pairwise Comparisons

Comparison of Rank Aggregation Methods Based on Inherent Ability

Multiple Query-Independent Values Based Asymmetric Ranking for Approximate Nearest Neighbor Search

STRank: A SiteRank Algorithm Using Semantic Relevance and Time Frequency

Ranking Fuzzy Numbers Based on the Divergence

A novel parallel distance metric-based approach for diversified ranking on large graphs.