Abstract:In the past two decades, psychological science has experienced an unprecedented replicability crisis which uncovered several issues. Among others, statistical inference is too often viewed as an isolated procedure limited to the analysis of data that have already been collected. We build on and further develop an idea proposed by Gelman and Carlin (2014) termed "prospective and retrospective design analysis". Rather than focusing only on the statistical significance of a result and on the classical control of type I and type II errors, a comprehensive design analysis involves reasoning about what can be considered a plausible effect size. Furthermore, it introduces two relevant inferential risks: the exaggeration ratio or Type M error (i.e., the predictable average overestimation of an effect that emerges as statistically significant), and the sign error or Type S error (i.e., the risk that a statistically significant effect is estimated in the wrong direction). Another important aspect of design analysis is that it can be usefully carried out both in the planning phase of a study and for the evaluation of studies that have already been conducted, thus increasing researchers' awareness during all phases of a research project. We use a familiar example in psychology where the researcher is interested in analyzing the differences between two independent groups. We examine the case in which the plausible effect size is formalized as a single value, and propose a method in which uncertainty concerning the magnitude of the effect is formalized via probability distributions. Through several examples, we show that even though a design analysis requires big effort, it has the potential to contribute to planning more robust and replicable studies. Finally, future developments in the Bayesian framework are discussed.

Flexible yet fair: blinding analyses in experimental psychology

The preregistration revolution

How to Produce, Identify, and Motivate Robust Psychological Science: A Roadmap and a Response to Vize et al.

Improving Transparency in Observational Social Science Research: A Pre-Analysis Plan Approach

Methods for applying blinding and randomisation in animal experiments

Campbell's Law Explains the Replication Crisis: Pre-Registration Badges Are History Repeating

Seeing the Truth About Double Blinding

Enhancing statistical inference in psychological research via prospective and retrospective design analysis

Scanning the horizon: towards transparent and reproducible neuroimaging research

The potential of preregistration in psychology: Assessing preregistration producibility and preregistration-study consistency.

Correcting for Bias in Psychology: A Comparison of Meta-Analytic Methods

Pre-registration for Predictive Modeling

Estimating the Prevalence of Transparency and Reproducibility-Related Research Practices in Psychology (2014–2017)

The same analysis approach: Practical protection against the pitfalls of novel neuroimaging analysis methods

How to be fair? A study of label and selection bias

A standardized framework to test event-based experiments

Auditing the AI auditors: A framework for evaluating fairness and bias in high stakes AI predictive models.

A simple statistical framework for small sample studies

An Excess of Positive Results: Comparing the Standard Psychology Literature With Registered Reports

Inspection-Guided Randomization: A Flexible and Transparent Restricted Randomization Framework for Better Experimental Design

Power Analysis and Effect Size in Mixed Effects Models: A Tutorial