Abstract:We study the problem of designing consistent sequential two-sample tests in a nonparametric setting. Guided by the principle of testing by betting, we reframe this task into that of selecting a sequence of payoff functions that maximize the wealth of a fictitious bettor, betting against the null in a repeated game. In this setting, the relative increase in the bettor’s wealth has a precise interpretation as the measure of evidence against the null, and thus our sequential test rejects the null when the wealth crosses an appropriate threshold. We develop a general framework for setting up the betting game for two-sample testing, in which the payoffs are selected by a prediction strategy as data-driven predictable estimates of the witness function associated with the variational representation of some statistical distance measures, such as integral probability metrics (IPMs). We then formally relate the statistical properties of the test (such as consistency, type-II error exponent and expected sample size) to the regret of the corresponding prediction strategy. We construct a practical sequential two-sample test by instantiating our general strategy with the kernel-MMD metric, and demonstrate its ability to adapt to the difficulty of the unknown alternative through theoretical and empirical results. Our framework is versatile, and easily extends to other problems; we illustrate this by applying our approach to construct consistent tests for the following problems: (i) time-varying two-sample testing with non-exchangeable observations, and (ii) an abstract class of “invariant” testing problems, including symmetry and independence testing.

A class of nonparametric tests for the two-sample problem based on order statistics

Non-parametric Tests for the Tail Equivalence Via Empirical Likelihood

The AUGUST Two-Sample Test: Powerful, Interpretable, and Fast

A class of Šidák-type tests based on maximal precedence and exceedance statistic

Permutation Tests at Nonparametric Rates

Testing distributional equality for functional random variables

A new class of robust two-sample Wald-type tests

A Kernel Method for the Two-Sample Problem

Two-Sample Smooth Tests for the Equality of Distributions

Nonparametric goodness of fit tests for Pareto type-I distribution with complete and censored data

On high-dimensional modifications of some graph-based two-sample tests

Log-Rank-Type Tests for Equality of Distributions in High-Dimensional Spaces.

A two-sample nonparametric test for one-sided location-scale alternative

An Entropy-Based Approach for Nonparametrically Testing Simple Probability Distribution Hypotheses

Nonparametric Two-Sample Testing by Betting

On a new class of tests for the Pareto distribution using Fourier methods

Sidak-type tests for the two-sample problem based on precedence and exceedance statistics

Two-Sample Test Based on Classification Probability

Generalized kernel two-sample tests

The New and Improved Two-Sample t Test

Bayesian Hypothesis Tests Using Nonparametric Statistics