Abstract:This paper describes the theoretical background, algorithm and validation of a recently developed novel method of ranking based on the sum of ranking differences [TrAC Trends Anal. Chem. 2010; 29: 101–109]. The ranking is intended to compare models, methods, analytical techniques, panel members, etc. and it is entirely general. First, the objects to be ranked are arranged in the rows and the variables (for example model results) in the columns of an input matrix. Then, the results of each model for each object are ranked in the order of increasing magnitude. The difference between the rank of the model results and the rank of the known, reference or standard results is then computed. (If the golden standard ranking is known the rank differences can be completed easily.) In the end, the absolute values of the differences are summed together for all models to be compared. The sum of ranking differences (SRD) arranges the models in a unique and unambiguous way. The closer the SRD value to zero (i.e. the closer the ranking to the golden standard), the better is the model. The proximity of SRD values shows similarity of the models, whereas large variation will imply dissimilarity. Generally, the average can be accepted as the golden standard in the absence of known or reference results, even if bias is also present in the model results in addition to random error. Validation of the SRD method can be carried out by using simulated random numbers for comparison (permutation test). A recursive algorithm calculates the discrete distribution for a small number of objects (n < 14), whereas the normal distribution is used as a reasonable approximation if the number of objects is large. The theoretical distribution is visualized for random numbers and can be used to identify SRD values for models that are far from being random. The ranking and validation procedures are called Sum of Ranking differences (SRD) and Comparison of Ranks by Random Numbers (CRNN), respectively. Copyright © 2010 John Wiley & Sons, Ltd. The theoretical background and algorithm are described for sum of ranking differences (SRD ‐ a novel procedure of ordering, grouping of methods, models). The proximity of SRD values shows similarity of the methods, whereas large variation will imply dissimilarity. Validation of the SRD procedure can be carried out by using simulated random numbers for comparison (a permutation test called CRRN). The theoretical distribution is visualized; probabilities are calculated to SRD values showing whether they are far from being random.

RGA: a unified measure of predictive accuracy

Rank-based Robust Regression Method

Connecting population-level AUC and latent scale-invariant $R^2$ via Semiparametric Gaussian Copula and rank correlations

Decision Curve Analysis: a Technical Note

Regression Diagnostics meets Forecast Evaluation: Conditional Calibration, Reliability Diagrams, and Coefficient of Determination

Assessing the calibration of multivariate probabilistic forecasts

The coefficient of determination R-squared is more informative than SMAPE, MAE, MAPE, MSE and RMSE in regression analysis evaluation

Regions of Reliability in the Evaluation of Multivariate Probabilistic Forecasts

Ranking with Confidence for Large Scale Comparison Data

A better measure of relative prediction accuracy for model selection and model estimation

Ranking earthquake forecasts using proper scoring rules: Binary events in a low probability environment

Sum of ranking differences for method discrimination and its validation: comparison of ranks with random numbers

Nonparametric Assessment of Variable Selection and Ranking Algorithms

A Rank-based Robust Linear Regression Analysis and Diagnostic Methods

Random Bits Regression: a Strong General Predictor for Big Data

A Tale of Trust and Accuracy: Base vs. Instruct LLMs in RAG Systems

RWA Web: A Free, Comprehensive, Web-Based, and User-Friendly Tool for Relative Weight Analyses

Evaluating Probabilistic Forecasts with scoringRules

Assessing the accuracy of directional forecasts

Regression and Independence Based Variable Importance Measure.

Evaluating the Discrimination Ability of Proper Multivariate Scoring Rules