Comparing discriminating abilities of evaluation metrics in link prediction

Xinshan Jiao,Shuyan Wan,Qian Liu,Yilin Bi,Yan-Li Lee,En Xu,Dong Hao,Tao Zhou

2024-01-08

Abstract:Link prediction aims to predict the potential existence of links between two unconnected nodes within a network based on the known topological characteristics. Evaluation metrics are used to assess the effectiveness of algorithms in link prediction. The discriminating ability of these evaluation metrics is vitally important for accurately evaluating link prediction algorithms. In this study, we propose an artificial network model, based on which one can adjust a single parameter to monotonically and continuously turn the prediction accuracy of the specifically designed link prediction algorithm. Building upon this foundation, we show a framework to depict the effectiveness of evaluating metrics by focusing on their discriminating ability. Specifically, a quantitative comparison in the abilities of correctly discerning varying prediction accuracies was conducted encompassing nine evaluation metrics: Precision, Recall, F1-Measure, Matthews Correlation Coefficient (MCC), Balanced Precision (BP), the Area Under the receiver operating characteristic Curve (AUC), the Area Under the Precision-Recall curve (AUPR), Normalized Discounted Cumulative Gain (NDCG), and the Area Under the magnified ROC (AUC-mROC). The results indicate that the discriminating abilities of the three metrics, AUC, AUPR, and NDCG, are significantly higher than those of other metrics.

Social and Information Networks,Data Analysis, Statistics and Probability

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the discriminative ability of evaluation metrics in link prediction. Specifically, the author focuses on how to continuously and monotonically change the prediction accuracy of a specifically - designed link prediction algorithm by adjusting the noise intensity in the algorithm, and on this basis, constructs a framework to evaluate the ability of different evaluation metrics to distinguish different prediction accuracies. By comparing the performance of nine evaluation metrics (precision, recall, F1 - score, Matthews correlation coefficient, balanced precision, area under the receiver operating characteristic curve, area under the precision - recall curve, normalized discounted cumulative gain, and amplified area under the ROC curve), the paper explores the effectiveness and applicability of these evaluation metrics. The research results show that the discriminative abilities of the three evaluation metrics, AUC, AUPR and NDCG, are significantly higher than those of other metrics.