Abstract:Epidemiological analyses often rely on p-values, which have lost their classical insight. We reinforce the classical interpretation of p-values in randomized experiments, especially in settings with “big data” and consequently many tests. Fisher first introduced the concept of p-value by tying it to the need for randomization of units to treatments. He proposed that researchers assess the sharp null hypothesis by conducting a randomization test summarized by a p-value. First, one choses a suitable test statistic, S, and one calculates its observed value, say S*. Then, one constructs the S's distribution induced by the randomization under the null hypothesis. To obtain such a randomization distribution, one enumerates all possible treatment assignments (Ntotal) based on the assignment mechanism, and for each, one calculates the value of S that would have been observed with that assignment. The proportion of such values of S across the possible randomizations that are as large or larger than S* is the p-value. To illustrate, consider a randomized experiment with 20 participants and 3 treatments given to each in random order so there are six (=3*2*1) possible sequences of treatments. Whatever the test statistic, the minimum p-value that can be achieved equals 1/Ntotal (i.e., 1/(20*6)≈0.0083), which is achieved when S* is the largest of all possible values of S. Ntotal depends on the number of units and treatments. For example, Bonferroni adjustments (e.g., dividing the significance level by the number of tests), often used to “correct” for multiple testing in environmental studies with high-dimensional outcomes (e.g., methylation on 450,000 CpG sites), ignore the classical insight of randomization-based p-values, because it is applied to model-based p-values, which are not justified by the study design. Here, applying Bonferroni with thousands of tests would yield a nonsensical “corrected” significance level less than the minimal achievable p-value.

Valid Randomization‐based P‐values for Partially Post Hoc Subgroup Analyses

Evaluating the Validity of Post-Hoc Subgroup Inferences: A Case Study

High-Dimensional Randomized Crossover Studies: A Clarification of P-Values Interpretation

More powerful randomization‐based p‐values in double‐blind trials with non‐compliance

Randomized p-values for multiple testing and their application in replicability analysis

More Powerful Randomization-Basedp-values in Double-Blind Trials with Non-Compliance

Challenges and Solutions to Pre- and Post-Randomization Subgroup Analyses

A Utility-Based Design for Randomized Comparative Trials with Ordinal Outcomes and Prognostic Subgroups

More Powerful Multiple Testing in Randomized Experiments with Non-Compliance

Post-hoc Hypothesis Testing

Rank-transformed subsampling: inference for multiple data splitting and exchangeable p-values

Multiple testing of composite null hypotheses for discrete data using randomized $p$-values

When possible, report a Fisher-exact P value and display its underlying null randomization distribution

Post-hoc and Anytime Valid Permutation and Group Invariance Testing

Multiple testing of interval composite null hypotheses using randomized p-values

Dempster-Shafer P-values: Thoughts on an Alternative Approach for Multinomial Inference

High-dimensional Randomization-Based Inference Capitalizing on Classical Design and Modern Computing

A Bayesian approach to subgroup identification.

A p-value for Process Tracing and other N=1 Studies

Valid p-Values and Expectations of p-Values Revisited

Posterior Predictive Propensity Scores and P-Values