Abstract:Epidemiological analyses often rely on p-values, which have lost their classical insight. We reinforce the classical interpretation of p-values in randomized experiments, especially in settings with “big data” and consequently many tests. Fisher first introduced the concept of p-value by tying it to the need for randomization of units to treatments. He proposed that researchers assess the sharp null hypothesis by conducting a randomization test summarized by a p-value. First, one choses a suitable test statistic, S, and one calculates its observed value, say S*. Then, one constructs the S's distribution induced by the randomization under the null hypothesis. To obtain such a randomization distribution, one enumerates all possible treatment assignments (Ntotal) based on the assignment mechanism, and for each, one calculates the value of S that would have been observed with that assignment. The proportion of such values of S across the possible randomizations that are as large or larger than S* is the p-value. To illustrate, consider a randomized experiment with 20 participants and 3 treatments given to each in random order so there are six (=3*2*1) possible sequences of treatments. Whatever the test statistic, the minimum p-value that can be achieved equals 1/Ntotal (i.e., 1/(20*6)≈0.0083), which is achieved when S* is the largest of all possible values of S. Ntotal depends on the number of units and treatments. For example, Bonferroni adjustments (e.g., dividing the significance level by the number of tests), often used to “correct” for multiple testing in environmental studies with high-dimensional outcomes (e.g., methylation on 450,000 CpG sites), ignore the classical insight of randomization-based p-values, because it is applied to model-based p-values, which are not justified by the study design. Here, applying Bonferroni with thousands of tests would yield a nonsensical “corrected” significance level less than the minimal achievable p-value.

Counternull Sets in Randomized Experiments

The Counternull Value of an Effect Size - A New Statistic

When possible, report a Fisher-exact P value and display its underlying null randomization distribution

High-Dimensional Randomized Crossover Studies: A Clarification of P-Values Interpretation

Multiple testing of interval composite null hypotheses using randomized p-values

Multiple testing of composite null hypotheses for discrete data using randomized $p$-values

Beyond the Sharp Null: Randomization Inference, Bounded Null Hypotheses, and Confidence Intervals for Maximum Effects

On Some Assumptions of the Null Hypothesis Statistical Testing

More Powerful Randomization-Basedp-values in Double-Blind Trials with Non-Compliance

Randomized p-values for multiple testing and their application in replicability analysis

A p-value for Process Tracing and other N=1 Studies

High-dimensional Randomization-Based Inference Capitalizing on Classical Design and Modern Computing

Permutation p-values should never be zero: calculating exact p-values when permutations are randomly drawn

Bagged Empirical Null p-values: A Method to Account for Model Uncertainty in Large Scale Inference

Interval estimation, point estimation, and null hypothesis significance testing calibrated by an estimated posterior probability of the null hypothesis

Valid p-Values and Expectations of p-Values Revisited

Valid Randomization‐based P‐values for Partially Post Hoc Subgroup Analyses

Multiple testing using uniform filtering of ordered p-values

My Ban on Null Hypothesis Significance Testing and Confidence Intervals

Exact p-values for pairwise comparison of Friedman rank sums, with application to comparing classifiers

When More Is Less: Pitfalls of significance testing