Abstract:Much empirical science involves evaluating alternative explanations for the obtained data. For example, given certain assumptions underlying a statistical test, a "significant" result generally refers to implausibility of a null (zero) effect in the population producing the obtained study data. However, methodological work on various versions of p-hacking (i.e., using different analysis strategies until a "significant" result is produced) questions whether significant p-values might often reflect false findings. Indeed, initial simulations of single studies showed that the potential for finding "significant" but false findings might be much higher than the nominal .05 value when various analysis flexibilities are undertaken. In many settings, however, research articles report multiple studies using consistent methods across the studies, where those consistent methods would constrain the flexibilities used to produce high false-finding rates for simulations of single studies. Thus, we conducted simulations of study sets. These simulations show that consistent methods across studies (i.e., consistent in terms of which measures are analyzed, which conditions are included, and whether and how covariates are included) dramatically reduce the potential for flexible research practices (p-hacking) to produce consistent sets of significant results across studies. For p-hacking to produce even modest probabilities of a consistent set of studies would require (a) a large amount of selectivity in study reporting and (b) severe (and quite intentional) versions of p-hacking. With no more than modest selective reporting and with consistent methods across studies, p-hacking does not provide a plausible explanation for consistent empirical results across studies, especially as the size of the reported study set increases. In addition, the simulations show that p-hacking can produce high rates of false findings for single studies with very large samples. In contrast, a series of methodologically-consistent studies (even with much smaller samples) is much less vulnerable to the forms of p-hacking examined in the simulations.

Detecting p-hacking

p -Hacking, Data type and Data-Sharing Policy

The Extent and Consequences of P-Hacking in Science

Methods Matter: p-Hacking and Publication Bias in Causal Analysis in Economics: Comment

Post-hoc Hypothesis Testing

Multiple testing of composite null hypotheses for discrete data using randomized $p$-values

P‐hacking in meta‐analyses: A formalization and new meta‐analytic methods

Screening $p$-Hackers: Dissemination Noise as Bait

P-hacking in Academic Research : Evidence from Experimental Accounting Studies

Accumulating evidence across studies: Consistent methods protect against false findings produced by p-hacking

Randomized p-values for multiple testing and their application in replicability analysis

Multiple testing of interval composite null hypotheses using randomized p-values

Accurate and Ultra-Efficient p-Value Calculation for Higher Criticism Tests

Robustness of multiple testing procedures against dependence

An Entropy-Based Approach for Nonparametrically Testing Simple Probability Distribution Hypotheses

Examining publication bias—a simulation-based evaluation of statistical tests on publication bias

Detecting rare and weak deviations of non-proportional hazard in survival analysis

Distribution-free Multiple Testing

High-Dimensional Randomized Crossover Studies: A Clarification of P-Values Interpretation

A Change-Point Approach to Estimating the Proportion of False Null Hypotheses in Multiple Testing

A p-value for Process Tracing and other N=1 Studies