Abstract:We study the problem of designing consistent sequential two-sample tests in a nonparametric setting. Guided by the principle of testing by betting, we reframe this task into that of selecting a sequence of payoff functions that maximize the wealth of a fictitious bettor, betting against the null in a repeated game. In this setting, the relative increase in the bettor’s wealth has a precise interpretation as the measure of evidence against the null, and thus our sequential test rejects the null when the wealth crosses an appropriate threshold. We develop a general framework for setting up the betting game for two-sample testing, in which the payoffs are selected by a prediction strategy as data-driven predictable estimates of the witness function associated with the variational representation of some statistical distance measures, such as integral probability metrics (IPMs). We then formally relate the statistical properties of the test (such as consistency, type-II error exponent and expected sample size) to the regret of the corresponding prediction strategy. We construct a practical sequential two-sample test by instantiating our general strategy with the kernel-MMD metric, and demonstrate its ability to adapt to the difficulty of the unknown alternative through theoretical and empirical results. Our framework is versatile, and easily extends to other problems; we illustrate this by applying our approach to construct consistent tests for the following problems: (i) time-varying two-sample testing with non-exchangeable observations, and (ii) an abstract class of “invariant” testing problems, including symmetry and independence testing.

Sidak-type tests for the two-sample problem based on precedence and exceedance statistics

A class of Šidák-type tests based on maximal precedence and exceedance statistic

Non-parametric Tests for the Tail Equivalence Via Empirical Likelihood

A class of nonparametric tests for the two-sample problem based on order statistics

Nonparametric goodness of fit tests for Pareto type-I distribution with complete and censored data

Likelihood ratio type two-sample tests for current status data

Approximative Tests for the Equality of Two Cumulative Incidence Functions of a Competing Risk

Higher-Order Asymptotics and Its Application to Testing the Equality of the Examinee Ability Over Two Sets of Items

The AUGUST Two-Sample Test: Powerful, Interpretable, and Fast

A two-sample nonparametric test for one-sided location-scale alternative

Nonparametric Two-Sample Testing by Betting

Optimal exact tests for composite alternative hypotheses on cross tabulated data

Sequential Tests of Statistical Hypotheses with Confidence Limits

Active Sequential Two-Sample Testing

Sequential Predictive Two-Sample and Independence Testing

Graph-Based Tests for Two-Sample Comparisons of Categorical Data

An efficient asymptotic approach for testing monotone proportions assuming an underlying logit based order dose-response model

Exponentially Consistent Kernel Two-Sample Tests

Testing against ordered alternatives in one-way ANOVA model with exponential errors

A class of sequential multi-hypothesis tests

Log-Rank-Type Tests for Equality of Distributions in High-Dimensional Spaces.