Abstract:Non-parametric two-sample tests based on energy distance or maximum mean discrepancy are widely used statistical tests for comparing multivariate data from two populations. While these tests enjoy desirable statistical properties, their test statistics can be expensive to compute as they require the computation of 3 distinct Euclidean distance (or kernel) matrices between samples, where the time complexity of each of these computations (namely, $O(n_{x}^2 p)$, $O(n_{y}^2 p)$, and $O(n_{x} n_{y} p)$) scales quadratically with the number of samples ($n_x$, $n_y$) and linearly with the number of variables ($p$). Since the standard permutation test requires repeated re-computations of these expensive statistics it's application to large datasets can become unfeasible. While several statistical approaches have been proposed to mitigate this issue, they all sacrifice desirable statistical properties to decrease the computational cost (e.g., trade computation speed by a decrease in statistical power). A better computational strategy is to first pre-compute the Euclidean distance (kernel) matrix of the concatenated data, and then permute indexes and retrieve the corresponding elements to compute the re-sampled statistics. While this strategy can reduce the computation cost relative to the standard permutation test, it relies on the computation of a larger Euclidean distance (kernel) matrix with complexity $O((n_x + n_y)^2 p)$. In this paper, we present a novel computationally efficient permutation algorithm which only requires the pre-computation of the 3 smaller matrices and achieves large computational speedups without sacrificing finite-sample validity or statistical power. We illustrate its computational gains in a series of experiments and compare its statistical power to the current state-of-the-art approach for balancing computational cost and statistical performance.

The Classification Permutation Test: A Nonparametric Test for Equality of Multivariate Distributions

An Empirical Comparison of Parametric and Permutation Tests for Regression Analysis of Randomized Experiments

Permutation Tests at Nonparametric Rates

Kernel-Based Partial Permutation Test for Detecting Heterogeneous Functional Relationship

The conditional permutation test for independence while controlling for confounders

Multivariate quantile-based permutation tests with application to functional data

Permutation Tests for Assessing Potential Non-Linear Associations between Treatment Use and Multivariate Clinical Outcomes

A Permutation Test for Assessing the Presence of Individual Differences in Treatment Effects

Permutation tests for the equality of covariance operators of functional data with applications to evolutionary biology

Functional Response Designs via the Analytic Permutation Test

PERMUTOOLS: A MATLAB Package for Multivariate Permutation Testing

Fast permutation tests and related methods, for association between rare variants and binary outcomes

Consistency of invariance-based randomization tests

A PERMUTATION TEST FOR TWO-SAMPLE MEANS AND SIGNAL IDENTIFICATION OF HIGH-DIMENSIONAL DATA

Randomization Tests that Condition on Non-Categorical Covariate Balance

Residual Permutation Test for High-Dimensional Regression Coefficient Testing

Least Squares-Based Permutation Tests in Time Series

Permutation tests for detecting and estimating mixtures in task performance within groups.

Computationally efficient permutation tests for the multivariate two-sample problem based on energy distance or maximum mean discrepancy statistics

Another look at the Lady Tasting Tea and differences between permutation tests and randomization tests

Equivalence Test in Multi-dimensional Space with Applications in A/B Testing