Abstract:In the past two decades, psychological science has experienced an unprecedented replicability crisis which uncovered several issues. Among others, statistical inference is too often viewed as an isolated procedure limited to the analysis of data that have already been collected. We build on and further develop an idea proposed by Gelman and Carlin (2014) termed "prospective and retrospective design analysis". Rather than focusing only on the statistical significance of a result and on the classical control of type I and type II errors, a comprehensive design analysis involves reasoning about what can be considered a plausible effect size. Furthermore, it introduces two relevant inferential risks: the exaggeration ratio or Type M error (i.e., the predictable average overestimation of an effect that emerges as statistically significant), and the sign error or Type S error (i.e., the risk that a statistically significant effect is estimated in the wrong direction). Another important aspect of design analysis is that it can be usefully carried out both in the planning phase of a study and for the evaluation of studies that have already been conducted, thus increasing researchers' awareness during all phases of a research project. We use a familiar example in psychology where the researcher is interested in analyzing the differences between two independent groups. We examine the case in which the plausible effect size is formalized as a single value, and propose a method in which uncertainty concerning the magnitude of the effect is formalized via probability distributions. Through several examples, we show that even though a design analysis requires big effort, it has the potential to contribute to planning more robust and replicable studies. Finally, future developments in the Bayesian framework are discussed.

Retrospective-prospective symmetry in the likelihood and Bayesian analysis of case-control studies

Retrospective Versus Prospective Score Tests For Genetic Association With Case-Control Data

Modified Poisson Regression Model for Data of Prospective Studies with Common Outcomes

Analysis of Case-Control Association Studies: SNPs, Imputation and Haplotypes

CASE-CONTROL SURVIVAL ANALYSIS WITH A GENERAL SEMIPARAMETRIC SHARED FRAILTY MODEL - A PSEUDO FULL LIKELIHOOD APPROACH

Unlocking Retrospective Prevalent Information in EHRs -- a Pairwise Pseudolikelihood Approach

Retrospective Likelihood-Based Methods for Analyzing Case-Cohort Genetic Association Studies.

Prospective and Retrospective Causal Inferences Based on the Potential Outcome Framework

Semiparametric maximum likelihood estimation with two-phase stratified case-control sampling

A Comparison of the Polytomous Logistic Regression and Joint Cox Proportional Hazards Models for Evaluating Multiple Disease Subtypes in Prospective Cohort Studies

Statistical inference for case-control logistic regression via integrating external summary data

A bias-reduced generalized estimating equation approach for proportional odds models with small-sample longitudinal ordinal data

Modern Bayesian Statistics in Clinical Research

Enhancing statistical inference in psychological research via prospective and retrospective design analysis

Bayesian joint analysis of heterogeneous- and skewed-longitudinal data and a binary outcome, with application to AIDS clinical studies

Case-control studies for rare diseases: improved estimation of several risks and of feature dependences

The Fragility Of Standard Inferential Approaches In Principal Stratification Models Relative To Direct Likelihood Approaches

Accounting for selection biases in population analyses: equivalence of the in-likelihood and post-processing approaches

On partial likelihood

Variance estimation for logistic regression in case-cohort studies

A composite likelihood ratio approach to the analysis of correlated binary data in genetic association studies