Abstract:Much empirical science involves evaluating alternative explanations for the obtained data. For example, given certain assumptions underlying a statistical test, a "significant" result generally refers to implausibility of a null (zero) effect in the population producing the obtained study data. However, methodological work on various versions of p-hacking (i.e., using different analysis strategies until a "significant" result is produced) questions whether significant p-values might often reflect false findings. Indeed, initial simulations of single studies showed that the potential for finding "significant" but false findings might be much higher than the nominal .05 value when various analysis flexibilities are undertaken. In many settings, however, research articles report multiple studies using consistent methods across the studies, where those consistent methods would constrain the flexibilities used to produce high false-finding rates for simulations of single studies. Thus, we conducted simulations of study sets. These simulations show that consistent methods across studies (i.e., consistent in terms of which measures are analyzed, which conditions are included, and whether and how covariates are included) dramatically reduce the potential for flexible research practices (p-hacking) to produce consistent sets of significant results across studies. For p-hacking to produce even modest probabilities of a consistent set of studies would require (a) a large amount of selectivity in study reporting and (b) severe (and quite intentional) versions of p-hacking. With no more than modest selective reporting and with consistent methods across studies, p-hacking does not provide a plausible explanation for consistent empirical results across studies, especially as the size of the reported study set increases. In addition, the simulations show that p-hacking can produce high rates of false findings for single studies with very large samples. In contrast, a series of methodologically-consistent studies (even with much smaller samples) is much less vulnerable to the forms of p-hacking examined in the simulations.

P-hacking in Academic Research : Evidence from Experimental Accounting Studies

P-Hacking in Experimental Accounting Studies

p -Hacking, Data type and Data-Sharing Policy

Discontinuous Distribution of Test Statistics Around Significance Thresholds in Empirical Accounting Studies

Accumulating evidence across studies: Consistent methods protect against false findings produced by p-hacking

Methods Matter: p-Hacking and Publication Bias in Causal Analysis in Economics: Comment

The Search for Significance: A Few Peculiarities in the Distribution of P Values in Experimental Psychology Literature

Detecting p-hacking

Study on Initial Audit Pricing Discount of China's Public Companies:Evidence from Auditor Switch

Everything hacked? What is the evidential value of the experimental public administration literature?

Are Accounting Estimate Changes Associated with an Aggressive Earnings Effect? Empirical Evidence

Estimating the extent of selective reporting: An application to economics

Why There Are So Many Contradicted or Exaggerated Findings in Highly-Cited Clinical Research?

Sixty Seconds on . . . P-hacking

P-value: A Bless or A Curse for Evidence-Based Studies?

Selective reporting of placebo tests in top economics journals

Detecting and Predicting Accounting Irregularities: A Comparison of Commercial and Academic Risk Measures

P-hacking in clinical trials and how incentives shape the distribution of results across phases

Changes in methodological study characteristics in psychology between 2010-2021

Can We Mathematically Spot Possible Manipulation of Results in Research Manuscripts Using Benford's Law?

Propensity Score Matching in Accounting Research