Abstract:We propose a kernel-based partial permutation test for checking the equality of functional relationship between response and covariates among different groups. The main idea, which is intuitive and easy to implement, is to keep the projections of the response vector Y on leading principle components of a kernel matrix fixed and permute Y ’s projections on the remaining principle components. The proposed test allows for different choices of kernels, corresponding to different classes of functions under the null hypothesis. First, using linear or polynomial kernels, our partial permutation tests are exactly valid in finite samples for linear or polynomial regression models with Gaussian noise; similar results straightforwardly extend to kernels with finite feature spaces. Second, by allowing the kernel feature space to diverge with the sample size, the test can be large-sample valid for a wider class of functions. Third, for general kernels with possibly infinite-dimensional feature space, the partial permutation test is exactly valid when the covariates are exactly balanced across all groups, or asymptotically valid when the underlying function follows certain regularized Gaussian processes. We further suggest test statistics using likelihood ratio between two (nested) Gaussian process regression models, and propose computationally efficient algorithms utilizing the EM algorithm and Newton’s method, where the latter also involves Fisher scoring and quadratic programming and is particularly useful when EM suffers from slow convergence. Extensions to correlated and non-Gaussian noises have also been investigated theoretically or numerically. Furthermore, the test can be extended to use multiple kernels together and can thus enjoy properties from each kernel. Both simulation study and application illustrate the properties of the proposed test.

Permutation Test for Validation of Binary Regression Models

Generalized Permutation Framework for Testing Model Variable Significance

An Empirical Comparison of Parametric and Permutation Tests for Regression Analysis of Randomized Experiments

Testing for no effect in regression problems: A permutation approach

Permutation Tests for Two-Sample Means of High-dimensional Data

A PERMUTATION TEST FOR TWO-SAMPLE MEANS AND SIGNAL IDENTIFICATION OF HIGH-DIMENSIONAL DATA

Kernel-Based Partial Permutation Test for Detecting Heterogeneous Functional Relationship

Permutation-based multiple testing when fitting many generalized linear models

The Classification Permutation Test: A Nonparametric Test for Equality of Multivariate Distributions

Overview of model validation for survival regression model with competing risks using melanoma study data

The Exchangeability Assumption for Permutation Tests of Multiple Regression Models: Implications for Statistics and Data Science

Impact Analysis of Classification Performance for Cross-Validation of Imbalance Spliting Data

Least Squares-Based Permutation Tests in Time Series

Blocked 3×2 Cross-Validated T-Test for Comparing Supervised Classification Learning Algorithms

Logistic Regression with Misclassification in Binary Outcome Variables: a Method and Software

Residual Permutation Test for High-Dimensional Regression Coefficient Testing

Block-regularized 5$\times$2 Cross-validated McNemar's Test for Comparing Two Classification Algorithms

Hypothesis Tests in Ordinal Predictive Models with Optimal Accuracy.

Permutation Tests for Assessing Potential Non-Linear Associations between Treatment Use and Multivariate Clinical Outcomes

Statistical Methods for Comparing Predictive Values in Medical Diagnosis