Abstract:BackgroundThe Friedman rank sum test is a widely-used nonparametric method in computational biology. In addition to examining the overall null hypothesis of no significant difference among any of the rank sums, it is typically of interest to conduct pairwise comparison tests. Current approaches to such tests rely on large-sample approximations, due to the numerical complexity of computing the exact distribution. These approximate methods lead to inaccurate estimates in the tail of the distribution, which is most relevant for p-value calculation.ResultsWe propose an efficient, combinatorial exact approach for calculating the probability mass distribution of the rank sum difference statistic for pairwise comparison of Friedman rank sums, and compare exact results with recommended asymptotic approximations. Whereas the chi-squared approximation performs inferiorly to exact computation overall, others, particularly the normal, perform well, except for the extreme tail. Hence exact calculation offers an improvement when small p-values occur following multiple testing correction. Exact inference also enhances the identification of significant differences whenever the observed values are close to the approximate critical value. We illustrate the proposed method in the context of biological machine learning, were Friedman rank sum difference tests are commonly used for the comparison of classifiers over multiple datasets.ConclusionsWe provide a computationally fast method to determine the exact p-value of the absolute rank sum difference of a pair of Friedman rank sums, making asymptotic tests obsolete. Calculation of exact p-values is easy to implement in statistical software and the implementation in R is provided in one of the Additional files and is also available at http://www.ru.nl/publish/pages/726696/friedmanrsd.zip.

R. A. Fisher's Exact Test Revisited

A Computer Program for Fisher's Exact Probability Test

Efficient three-stage $t$-tests

A Rapid FORTRAN Subroutine for the Fisher Exact Probability Test

Use of the Fisher Exact Probability Test in surface-to-air missile (SAM) countermeasures evaluations

Another look at the Lady Tasting Tea and differences between permutation tests and randomization tests

A two-stage Fisher exact test for multi-arm studies with binary outcome variables

Continuity Correction for Fisher's Exact Probability Test

Te Test: A New Non-asymptotic T-test for Behrens-Fisher Problems

Beyond the Sharp Null: Randomization Inference, Bounded Null Hypotheses, and Confidence Intervals for Maximum Effects

Fisher exact scanning for dependency

Is Fisher inference inferior to Neyman inference for policy analysis?

Subroutines for Computing Exact CHI-Square and Fisher's Exact Probability Tests

Exact p-values for pairwise comparison of Friedman rank sums, with application to comparing classifiers

A Program for Computing Fisher's Exact Probability Test and the Coefficient of Association λ for n × m Contingency Tables

When possible, report a Fisher-exact P value and display its underlying null randomization distribution

Testing the fit of data and external sets via an imprecise Sargan-Hansen test

A randomization-based perspective of analysis of variance: a test statistic robust to treatment effect heterogeneity

Comments on the Neyman-Fisher Controversy and Its Consequences

Fisher, Neyman-Pearson or NHST? A Tutorial for Teaching Data Testing

Significance testing without truth