Abstract:We study the problem of designing consistent sequential two-sample tests in a nonparametric setting. Guided by the principle of testing by betting, we reframe this task into that of selecting a sequence of payoff functions that maximize the wealth of a fictitious bettor, betting against the null in a repeated game. In this setting, the relative increase in the bettor’s wealth has a precise interpretation as the measure of evidence against the null, and thus our sequential test rejects the null when the wealth crosses an appropriate threshold. We develop a general framework for setting up the betting game for two-sample testing, in which the payoffs are selected by a prediction strategy as data-driven predictable estimates of the witness function associated with the variational representation of some statistical distance measures, such as integral probability metrics (IPMs). We then formally relate the statistical properties of the test (such as consistency, type-II error exponent and expected sample size) to the regret of the corresponding prediction strategy. We construct a practical sequential two-sample test by instantiating our general strategy with the kernel-MMD metric, and demonstrate its ability to adapt to the difficulty of the unknown alternative through theoretical and empirical results. Our framework is versatile, and easily extends to other problems; we illustrate this by applying our approach to construct consistent tests for the following problems: (i) time-varying two-sample testing with non-exchangeable observations, and (ii) an abstract class of “invariant” testing problems, including symmetry and independence testing.

The Bayesian Two-Sample t Test

Bayesian two-sample tests

A two-sample Bayesian t-test for microarray data

Bayesian Hypothesis Tests Using Nonparametric Statistics

A frequentist two-sample test based on Bayesian model selection

Bayesian tests of two proportions: A tutorial with R and JASP

Model-averaged Bayesian t tests

A new Bayesian two-sample t-test for effect size estimation under uncertainty based on a two-component Gaussian mixture with known allocations and the region of practical equivalence

Why bother with Bayesian t-tests?

Two Small Samples Bayesian Test Based on Conjugate Prior Distribution and the Instance Analysis

The New and Improved Two-Sample t Test

An Objective and Robust Bayes Factor for the Hypothesis Test One Sample and Two Population Means

Reference and Probability-Matching Priors for the Parameters of a Univariate Student $t$-Distribution

Bayesian Data Analysis: A Fresh Approach to Power Issues and Null Hypothesis Interpretation

On Bayes factors for hypothesis tests

A new approach to Bayesian hypothesis testing

Two-Sample Test Based on Classification Probability

Nonparametric Two-Sample Testing by Betting

A puzzle of proportions: Two popular Bayesian tests can yield dramatically different conclusions

A Non-parametric Test for the Two-Sample Problem

Posterior-based Wald-type Statistics for Hypothesis Testing