Abstract:Confronted with the challenge of identifying the most suitable metric to validate the merits of newly proposed models, the decision-making process is anything but straightforward. Given that comparing rankings introduces its own set of formidable challenges and the likely absence of a universal metric applicable to all scenarios, the scenario does not get any better. Furthermore, metrics designed for specific contexts, such as for Recommender Systems, sometimes extend to other domains without a comprehensive grasp of their underlying mechanisms, resulting in unforeseen outcomes and potential misuses. Complicating matters further, distinct metrics may emphasize different aspects of rankings, frequently leading to seemingly contradictory comparisons of model results and hindering the trustworthiness of evaluations. We unveil these aspects in the domain of ranking evaluation metrics. Firstly, we show instances resulting in inconsistent evaluations, sources of potential mistrust in commonly used metrics; by quantifying the frequency of such disagreements, we prove that these are common in rankings. Afterward, we conceptualize rankings using the mathematical formalism of symmetric groups detaching from possible domains where the metrics have been created; through this approach, we can rigorously and formally establish essential mathematical properties for ranking evaluation metrics, essential for a deeper comprehension of the source of inconsistent evaluations. We conclude with a discussion, connecting our theoretical analysis to the practical applications, highlighting which properties are important in each domain where rankings are commonly evaluated. In conclusion, our analysis sheds light on ranking evaluation metrics, highlighting that inconsistent evaluations should not be seen as a source of mistrust but as the need to carefully choose how to evaluate our models in the future.

The Measures of Rank or Status: A Reformulation and Reinterpretation

Scalable and Axiomatic Ranking of Network Role Similarity

Exponential-family Random Graph Models for Rank-order Relational Data

A physical model for efficient ranking in networks

Perception, Prestige and PageRank

The gap between public evaluations and interactional status in social networks

Reverse Ranking by Graph Structure: Model and Scalable Algorithms

A model for efficient dynamical ranking in networks

A Reputation-Based Model for Decision-Making in Online Social Networks

E-rank: A Structural-Based Similarity Measure in Social Networks.

Measuring Prestige in Online Social Networks

Ranking evaluation metrics from a group-theoretic perspective

Active influence in dynamical models of structural balance in social networks

Model for efficient dynamical ranking in networks

Inferring Social Roles and Statuses in Social Networks

An Influence Network Model to Study Discrepancies in Expressed and Private Opinions

Ranking the Importance of Nodes of Complex Networks by the Equivalence Classes Approach

Ranking of nodes of networks taking into account the power function of its weight of connections

Measuring tie strength in implicit social networks

An Enhanced Gravity Model for Determining Crucial Nodes in Social Networks Based on Degree K-Shell Eigenvector Index

Analysis of the Formation of the Structure of Social Networks using Latent Space Models for Ranked Dynamic Networks