Abstract:Epidemiological analyses often rely on p-values, which have lost their classical insight. We reinforce the classical interpretation of p-values in randomized experiments, especially in settings with “big data” and consequently many tests. Fisher first introduced the concept of p-value by tying it to the need for randomization of units to treatments. He proposed that researchers assess the sharp null hypothesis by conducting a randomization test summarized by a p-value. First, one choses a suitable test statistic, S, and one calculates its observed value, say S*. Then, one constructs the S's distribution induced by the randomization under the null hypothesis. To obtain such a randomization distribution, one enumerates all possible treatment assignments (Ntotal) based on the assignment mechanism, and for each, one calculates the value of S that would have been observed with that assignment. The proportion of such values of S across the possible randomizations that are as large or larger than S* is the p-value. To illustrate, consider a randomized experiment with 20 participants and 3 treatments given to each in random order so there are six (=3*2*1) possible sequences of treatments. Whatever the test statistic, the minimum p-value that can be achieved equals 1/Ntotal (i.e., 1/(20*6)≈0.0083), which is achieved when S* is the largest of all possible values of S. Ntotal depends on the number of units and treatments. For example, Bonferroni adjustments (e.g., dividing the significance level by the number of tests), often used to “correct” for multiple testing in environmental studies with high-dimensional outcomes (e.g., methylation on 450,000 CpG sites), ignore the classical insight of randomization-based p-values, because it is applied to model-based p-values, which are not justified by the study design. Here, applying Bonferroni with thousands of tests would yield a nonsensical “corrected” significance level less than the minimal achievable p-value.

P-value calibration for multiple testing problems in genomics

A statistical method for the conservative adjustment of false discovery rate (q-value)

A Practitioner's Guide to Multiple Testing Error Rates

Calibration of P-values for calibration and for deviation of a subpopulation from the full population

Smaller $p$-values in genomics studies using distilled historical information

Multiple testing when many $p$-values are uniformly conservative, with application to testing qualitative interaction in educational interventions

A more practical approach for the Benjamini-Hochberg FDR controlling procedure for huge-scale testing problems

Bagged Empirical Null p-values: A Method to Account for Model Uncertainty in Large Scale Inference

Multiple tests of association with biological annotation metadata

High-Dimensional Randomized Crossover Studies: A Clarification of P-Values Interpretation

A New Multiple Testing Method in the Dependent Case

Multiple testing using uniform filtering of ordered p-values

Simple solution to a common statistical problem: Interpreting multiple tests

MultipleTesting.com: a tool for life science researchers for multiple hypothesis testing correction

Evaluating the effective numbers of independent tests and significant p-value thresholds in commercial genotyping arrays and public imputation reference datasets

Testing hypotheses on a tree: new error rates and controlling strategies

Multiple testing of composite null hypotheses for discrete data using randomized $p$-values

A Change-Point Approach to Estimating the Proportion of False Null Hypotheses in Multiple Testing

Estimating the null distribution for conditional inference and genome-scale screening

Multiple testing of interval composite null hypotheses using randomized p-values

Selective testing and its effect on false discovery rate controlling procedures under discrete framework