Abstract:BackgroundThe Friedman rank sum test is a widely-used nonparametric method in computational biology. In addition to examining the overall null hypothesis of no significant difference among any of the rank sums, it is typically of interest to conduct pairwise comparison tests. Current approaches to such tests rely on large-sample approximations, due to the numerical complexity of computing the exact distribution. These approximate methods lead to inaccurate estimates in the tail of the distribution, which is most relevant for p-value calculation.ResultsWe propose an efficient, combinatorial exact approach for calculating the probability mass distribution of the rank sum difference statistic for pairwise comparison of Friedman rank sums, and compare exact results with recommended asymptotic approximations. Whereas the chi-squared approximation performs inferiorly to exact computation overall, others, particularly the normal, perform well, except for the extreme tail. Hence exact calculation offers an improvement when small p-values occur following multiple testing correction. Exact inference also enhances the identification of significant differences whenever the observed values are close to the approximate critical value. We illustrate the proposed method in the context of biological machine learning, were Friedman rank sum difference tests are commonly used for the comparison of classifiers over multiple datasets.ConclusionsWe provide a computationally fast method to determine the exact p-value of the absolute rank sum difference of a pair of Friedman rank sums, making asymptotic tests obsolete. Calculation of exact p-values is easy to implement in statistical software and the implementation in R is provided in one of the Additional files and is also available at http://www.ru.nl/publish/pages/726696/friedmanrsd.zip.

A Simple Algorithm for Exact Multinomial Tests

Optimal exact tests for multiple binary endpoints

Exact Multivariate Tests - A New Effective Principle of Controlled Model Choice

Exact p-values for pairwise comparison of Friedman rank sums, with application to comparing classifiers

Subroutines for Computing Exact CHI-Square and Fisher's Exact Probability Tests

Exact and efficient multivariate two-sample tests through generalized linear rank statistics

Exact conditional p-values from arbitrary ranking of a sample space: An application to genome-wide association studies

Testing high-dimensional multinomials with applications to text analysis

Towards a theory for testing statistical hypothesis: Multivariate mean with nuisance covariance matrix

A generalized correlated binomial distribution with application in multiple testing problems

Multiple Testing with Heterogeneous Multinomial Distributions

Two-Sample Test for Sparse High Dimensional Multinomial Distributions

A New Multivariate Test Formulation: Theory, Implementation, and Applications to Genome-Scale Sequencing and Expression

An Entropy-Based Approach for Nonparametrically Testing Simple Probability Distribution Hypotheses

A Program for Computing Fisher's Exact Probability Test and the Coefficient of Association λ for n × m Contingency Tables

Fast Approximation of Small P-values in Permutation Tests by Partitioning the Permutations

Exact Confidence Intervals for Linear Combinations of Multinomial Probabilities

An exact projection pursuit-based algorithm for multivariate two-sample nonparametric testing applicable to retrospective and group sequential studies

Discriminating Quantum States: the Multiple Chernoff Distance

Optimal E-Values for Exponential Families: the Simple Case

A minimum Wasserstein distance approach to Fisher's combination of independent discrete p-values