Abstract:BackgroundThe Friedman rank sum test is a widely-used nonparametric method in computational biology. In addition to examining the overall null hypothesis of no significant difference among any of the rank sums, it is typically of interest to conduct pairwise comparison tests. Current approaches to such tests rely on large-sample approximations, due to the numerical complexity of computing the exact distribution. These approximate methods lead to inaccurate estimates in the tail of the distribution, which is most relevant for p-value calculation.ResultsWe propose an efficient, combinatorial exact approach for calculating the probability mass distribution of the rank sum difference statistic for pairwise comparison of Friedman rank sums, and compare exact results with recommended asymptotic approximations. Whereas the chi-squared approximation performs inferiorly to exact computation overall, others, particularly the normal, perform well, except for the extreme tail. Hence exact calculation offers an improvement when small p-values occur following multiple testing correction. Exact inference also enhances the identification of significant differences whenever the observed values are close to the approximate critical value. We illustrate the proposed method in the context of biological machine learning, were Friedman rank sum difference tests are commonly used for the comparison of classifiers over multiple datasets.ConclusionsWe provide a computationally fast method to determine the exact p-value of the absolute rank sum difference of a pair of Friedman rank sums, making asymptotic tests obsolete. Calculation of exact p-values is easy to implement in statistical software and the implementation in R is provided in one of the Additional files and is also available at http://www.ru.nl/publish/pages/726696/friedmanrsd.zip.

Using Excel software to calculate Bayesian factors: taking goodness of fit test (Chi-square test) as an example

EBT: a Statistic Test Identifying Moderate Size of Significant Features with Balanced Power and Precision for Genome-Wide Rate Comparisons

A Good check on the Bayes factor

The Bayes Factor and Its Implementation in JASP: A Practical Primer

Bayesian Hypothesis Tests Using Nonparametric Statistics

A puzzle of proportions: Two popular Bayesian tests can yield dramatically different conclusions

Bayes factor testing of equality and order constraints on measures of association in social research

Bayes factor functions for reporting outcomes of hypothesis tests

Exact Test of Goodness of Fit for Binomial Distribution

On Bayes factors for hypothesis tests

Bayesian estimation of extropy and goodness of fit tests

An Objective and Robust Bayes Factor for the Hypothesis Test One Sample and Two Population Means

Bayesian tests of two proportions: A tutorial with R and JASP

A Bayesian Hierarchical Model for Comparing Average F1 Scores

Closed-Form Power and Sample Size Calculations for Bayes Factors

Nonparametric Bayes Factors Based on Empirical Likelihood Ratios.

On the Null Distribution of Bayes Factors in Linear Regression

Exact p-values for pairwise comparison of Friedman rank sums, with application to comparing classifiers

Bayesian alternatives to null hypothesis significance testing in biomedical research: a non-technical introduction to Bayesian inference with JASP

Assessing Bayes Factor Surfaces Using Interactive Visualization and Computer Surrogate Modeling

A Goodness-of-Fit Test for Statistical Models