Abstract:Hypothesis tests calibrated by (re)sampling methods (such as permutation, rank and bootstrap tests) are useful tools for statistical analysis, at the computational cost of requiring Monte-Carlo sampling for calibration. It is common and almost universal practice to execute such tests with predetermined and large number of Monte-Carlo samples, and disregard any randomness from this sampling at the time of drawing and reporting inference. At best, this approach leads to computational inefficiency, and at worst to invalid inference. That being said, a number of approaches in the literature have been proposed to adaptively guide analysts in choosing the number of Monte-Carlo samples, by sequentially deciding when to stop collecting samples and draw inference. These works introduce varying competing notions of what constitutes "valid" inference, complicating the landscape for analysts seeking suitable methodology. Furthermore, the majority of these approaches solely guarantee a meaningful estimate of the testing outcome, not the $p$-value itself $\unicode{x2014}$ which is insufficient for many practical applications. In this paper, we survey the relevant literature, and build bridges between the scattered validity notions, highlighting some of their complementary roles. We also introduce a new practical methodology that provides an estimate of the $p$-value of the Monte-Carlo test, endowed with practically relevant validity guarantees. Moreover, our methodology is sequential, updating the $p$-value estimate after each new Monte-Carlo sample has been drawn, while retaining important validity guarantees regardless of the selected stopping time. We conclude this paper with a set of recommendations for the practitioner, both in terms of selection of methodology and manner of reporting results.

Game-theoretic statistics and safe anytime-valid inference

Anytime-Valid Generalized Universal Inference on Risk Minimizers

On universal inference in Gaussian mixture models

Regularized e-processes: anytime valid inference with knowledge-based efficiency gains

Two ways game-theoretic probability can improve data analysis

Anytime-valid sequential testing for elicitable functionals via supermartingales

What Are the Odds? Improving the foundations of Statistical Model Checking

Distribution-uniform anytime-valid sequential inference

Inference with Sequential Monte-Carlo Computation of $p$-values: Fast and Valid Approaches

Statistical inference and hypotheses testing of risk averse stochastic programs

AIVAT: A New Variance Reduction Technique for Agent Evaluation in Imperfect Information Games

Safe Probabilistic Invariance Verification for Stochastic Discrete-Time Dynamical Systems

Stochastic Omega-Regular Verification and Control with Supermartingales

Auditing Fairness by Betting

Nonparametric Two-Sample Testing by Betting

A Framework for Safe Probabilistic Invariance Verification of Stochastic Dynamical Systems

How to Gamble If You're In a Hurry

Safe Testing

An Experimental Design for Anytime-Valid Causal Inference on Multi-Armed Bandits

Information-Theoretic Safe Bayesian Optimization

Evidential Reasoning in a Network Usage Prediction Testbed