Abstract:BackgroundThe Friedman rank sum test is a widely-used nonparametric method in computational biology. In addition to examining the overall null hypothesis of no significant difference among any of the rank sums, it is typically of interest to conduct pairwise comparison tests. Current approaches to such tests rely on large-sample approximations, due to the numerical complexity of computing the exact distribution. These approximate methods lead to inaccurate estimates in the tail of the distribution, which is most relevant for p-value calculation.ResultsWe propose an efficient, combinatorial exact approach for calculating the probability mass distribution of the rank sum difference statistic for pairwise comparison of Friedman rank sums, and compare exact results with recommended asymptotic approximations. Whereas the chi-squared approximation performs inferiorly to exact computation overall, others, particularly the normal, perform well, except for the extreme tail. Hence exact calculation offers an improvement when small p-values occur following multiple testing correction. Exact inference also enhances the identification of significant differences whenever the observed values are close to the approximate critical value. We illustrate the proposed method in the context of biological machine learning, were Friedman rank sum difference tests are commonly used for the comparison of classifiers over multiple datasets.ConclusionsWe provide a computationally fast method to determine the exact p-value of the absolute rank sum difference of a pair of Friedman rank sums, making asymptotic tests obsolete. Calculation of exact p-values is easy to implement in statistical software and the implementation in R is provided in one of the Additional files and is also available at http://www.ru.nl/publish/pages/726696/friedmanrsd.zip.

On two-sided p-values for non-symmetric distributions

A nonparametric two-sample conditional distribution test

Dempster-Shafer P-values: Thoughts on an Alternative Approach for Multinomial Inference

Selective inference is easier with p-values

The case for balanced hypothesis tests and equal-tailed confidence intervals

Two-Sample Test for Sparse High Dimensional Multinomial Distributions

A class of nonparametric tests for the two-sample problem based on order statistics

Multiple testing of composite null hypotheses for discrete data using randomized $p$-values

Conditional symmetry test based on empirical characteristic function

Nonparametric Two-Sample Testing by Betting

p-Value as the Strength of Evidence Measured by Confidence Distribution

Intrinsic tests for the equality of two correlated proportions

P-value: A Bless or A Curse for Evidence-Based Studies?

Double verification for two‐sample covariance matrices test

On high-dimensional modifications of some graph-based two-sample tests

Log-Rank-Type Tests for Equality of Distributions in High-Dimensional Spaces.

General Frameworks for Conditional Two-Sample Testing

Heavy-tailed $p$-value combinations from the perspective of extreme value theory

Optimal two-sided tests and confidence regions for Conway–Maxwell–Poisson parameters

Exponentially Consistent Kernel Two-Sample Tests

Exact p-values for pairwise comparison of Friedman rank sums, with application to comparing classifiers