Abstract:Much empirical science involves evaluating alternative explanations for the obtained data. For example, given certain assumptions underlying a statistical test, a "significant" result generally refers to implausibility of a null (zero) effect in the population producing the obtained study data. However, methodological work on various versions of p-hacking (i.e., using different analysis strategies until a "significant" result is produced) questions whether significant p-values might often reflect false findings. Indeed, initial simulations of single studies showed that the potential for finding "significant" but false findings might be much higher than the nominal .05 value when various analysis flexibilities are undertaken. In many settings, however, research articles report multiple studies using consistent methods across the studies, where those consistent methods would constrain the flexibilities used to produce high false-finding rates for simulations of single studies. Thus, we conducted simulations of study sets. These simulations show that consistent methods across studies (i.e., consistent in terms of which measures are analyzed, which conditions are included, and whether and how covariates are included) dramatically reduce the potential for flexible research practices (p-hacking) to produce consistent sets of significant results across studies. For p-hacking to produce even modest probabilities of a consistent set of studies would require (a) a large amount of selectivity in study reporting and (b) severe (and quite intentional) versions of p-hacking. With no more than modest selective reporting and with consistent methods across studies, p-hacking does not provide a plausible explanation for consistent empirical results across studies, especially as the size of the reported study set increases. In addition, the simulations show that p-hacking can produce high rates of false findings for single studies with very large samples. In contrast, a series of methodologically-consistent studies (even with much smaller samples) is much less vulnerable to the forms of p-hacking examined in the simulations.

Methods Matter: p-Hacking and Publication Bias in Causal Analysis in Economics: Comment

Detecting p-hacking

p -Hacking, Data type and Data-Sharing Policy

P-hacking in Academic Research : Evidence from Experimental Accounting Studies

Examining publication bias—a simulation-based evaluation of statistical tests on publication bias

Selective reporting of placebo tests in top economics journals

P‐hacking in meta‐analyses: A formalization and new meta‐analytic methods

Footprint of publication selection bias on meta‐analyses in medicine, environmental sciences, psychology, and economics

Accumulating evidence across studies: Consistent methods protect against false findings produced by p-hacking

Sensitivity analysis for publication bias in meta‐analyses

Do t-Statistic Hurdles Need to be Raised?

Estimating the extent of selective reporting: An application to economics

Tantalus on the Road to Asymptopia

P-Hacking in Experimental Accounting Studies

A modified regression method to test publication bias in meta-analyses with binary outcomes

Are Difference-in-Differences Estimates Blowing Smoke? A Cautionary Tale From an Oft-Used Natural Experiment

A regression-based method for detecting publication bias in multivariate meta-analysis

Systematic Evaluation and Comparison of Statistical Tests for Publication Bias

The power functions of Begg's and Egger's tests for publication bias

What’s in a p? Reassessing best practices for conducting and reporting hypothesis-testing research

Recent Developments in Inference: Practicalities for Applied Economics