Abstract:Conjecture about the weak replicability in social sciences has made scholars eager to quantify the scale and scope of replication failure for a discipline. Yet small-scale manual replication methods alone are ill-suited to deal with this big data problem. Here, we conduct a discipline-wide replication census in science. Our sample ( N = 14,126 papers) covers nearly all papers published in the six top-tier Psychology journals over the past 20 y. Using a validated machine learning model that estimates a paper's likelihood of replication, we found evidence that both supports and refutes speculations drawn from a relatively small sample of manual replications. First, we find that a single overall replication rate of Psychology poorly captures the varying degree of replicability among subfields. Second, we find that replication rates are strongly correlated with research methods in all subfields. Experiments replicate at a significantly lower rate than do non-experimental studies. Third, we find that authors' cumulative publication number and citation impact are positively related to the likelihood of replication, while other proxies of research quality and rigor, such as an author's university prestige and a paper's citations, are unrelated to replicability. Finally, contrary to the ideal that media attention should cover replicable research, we find that media attention is positively related to the likelihood of replication failure. Our assessments of the scale and scope of replicability are important next steps toward broadly resolving issues of replicability.

Large Scale Replication Projects in Contemporary Psychological Research

A large-scale in silico replication of ecological and evolutionary studies

Replication and Reproducibility in Cross-Cultural Psychology

Estimating the reproducibility of psychological science

A discipline-wide investigation of the replicability of Psychology papers over the past two decades

Statistical Methods for Replicability Assessment

Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015

A glass half full interpretation of the replicability of psychological science

Conceptualizing and Evaluating Replication Across Domains of Behavioral Research

Challenging the N-Heuristic: Effect size, not sample size, predicts the replicability of psychological science

Methodological concerns underlying a lack of evidence for cultural heterogeneity in the replication of psychological effects

Replication Markets: Results, Lessons, Challenges and Opportunities in AI Replication

Addressing the “Replication Crisis”: Using Original Studies to Design Replication Studies with Appropriate Statistical Power

Replication is important for educational psychology: Recent developments and key issues

The Replication Crisis in Psychological Research:The Solution of It

A Validity-Based Framework for Understanding Replication in Psychology

Replications are informative, particularly when they fail

Research into evidence-based psychological interventions needs a stronger focus on replicability

Replicating is difficult but necessary, and methodology can help

Replication in music psychology

The Pipeline Project: Pre-Publication Independent Replications of A Single Laboratory'S Research Pipeline