Abstract:Much empirical science involves evaluating alternative explanations for the obtained data. For example, given certain assumptions underlying a statistical test, a "significant" result generally refers to implausibility of a null (zero) effect in the population producing the obtained study data. However, methodological work on various versions of p-hacking (i.e., using different analysis strategies until a "significant" result is produced) questions whether significant p-values might often reflect false findings. Indeed, initial simulations of single studies showed that the potential for finding "significant" but false findings might be much higher than the nominal .05 value when various analysis flexibilities are undertaken. In many settings, however, research articles report multiple studies using consistent methods across the studies, where those consistent methods would constrain the flexibilities used to produce high false-finding rates for simulations of single studies. Thus, we conducted simulations of study sets. These simulations show that consistent methods across studies (i.e., consistent in terms of which measures are analyzed, which conditions are included, and whether and how covariates are included) dramatically reduce the potential for flexible research practices (p-hacking) to produce consistent sets of significant results across studies. For p-hacking to produce even modest probabilities of a consistent set of studies would require (a) a large amount of selectivity in study reporting and (b) severe (and quite intentional) versions of p-hacking. With no more than modest selective reporting and with consistent methods across studies, p-hacking does not provide a plausible explanation for consistent empirical results across studies, especially as the size of the reported study set increases. In addition, the simulations show that p-hacking can produce high rates of false findings for single studies with very large samples. In contrast, a series of methodologically-consistent studies (even with much smaller samples) is much less vulnerable to the forms of p-hacking examined in the simulations.

The Extent and Consequences of P-Hacking in Science

Accumulating evidence across studies: Consistent methods protect against false findings produced by p-hacking

p -Hacking, Data type and Data-Sharing Policy

P-hacking in Academic Research : Evidence from Experimental Accounting Studies

P‐hacking in meta‐analyses: A formalization and new meta‐analytic methods

Detecting p-hacking

P-Hacking in Experimental Accounting Studies

Sixty Seconds on . . . P-hacking

Methods Matter: p-Hacking and Publication Bias in Causal Analysis in Economics: Comment

Estimating the extent of selective reporting: An application to economics

The Search for Significance: A Few Peculiarities in the Distribution of P Values in Experimental Psychology Literature

Screening $p$-Hackers: Dissemination Noise as Bait

Why There Are So Many Contradicted or Exaggerated Findings in Highly-Cited Clinical Research?

False-Positive Psychology

Understanding p-values and significance

Thou Shalt Not Reject the P-value

Trap of trends to statistical significance: likelihood of near significant P value becoming more significant with extra data

Evidence-based medicine or statistically manipulated medicine? Are we slaves to the P -value?

Correcting for Bias in Psychology: A Comparison of Meta-Analytic Methods

Statistical witchhunts: Science, justice & the p-value crisis

Can We Mathematically Spot Possible Manipulation of Results in Research Manuscripts Using Benford's Law?