Abstract:The reproducibility movement in psychology has resulted in numerous highly publicized instances of replication failures. The goal of the present work was to investigate people’s reactions to a psychology replication failure vs. success, and to test whether a failure elicits harsher reactions when the researcher is a woman vs. a man. We examined these questions in a pre-registered experiment with a working adult sample, a conceptual replication of that experiment with a student sample, and an analysis of data compiled and posted by a psychology researcher on their public weblog with the stated goal to improve research replicability by rank-ordering psychology researchers by their “estimated false discovery risk.” Participants in the experiments were randomly assigned to read a news article describing a successful vs. failed replication attempt of original work from a male vs. female psychological scientist, and then completed measures of researcher competence, likability, integrity, perceptions of the research, and behavioral intentions for future interactions with the researcher. In both working adult and student samples, analyses consistently yielded large main effects of replication outcome, but no interaction with researcher gender. Likewise, the coding of weblog data posted in July 2021 indicated that 66.3% of the researchers scrutinized were men and 33.8% were women, and their rank-ordering was not correlated with researcher gender. The lack of support for our pre-registered gender-replication hypothesis is, at first glance, encouraging for women researchers’ careers; however, the substantial effect sizes we observed for replication outcome underscore the tremendous negative impact the reproducibility movement can have on psychologists’ careers. We discuss the implications of such negative perceptions and the possible downstream consequences for women in the field that are essential for future study.

Replication is important for educational psychology: Recent developments and key issues

Replication Research and Special Education

Replication of Special Education Research

Replication and Reproducibility in Cross-Cultural Psychology

A discipline-wide investigation of the replicability of Psychology papers over the past two decades

Replications are informative, particularly when they fail

Are replications mainstream now? A comparison of support for replications expressed in the policies of social psychology journals in 2015 and 2022

Registered reports and replications: An ongoing Journal of School Psychology initiative

Are Psychology Journals Anti-replication? A Snapshot of Editorial Practices

Interdisciplinary lessons and recommendations for the evaluation of replicability in behavioral sciences

Large Scale Replication Projects in Contemporary Psychological Research

Replication, falsification, and the crisis of confidence in social psychology

Giving Away the Science of School Psychology

Replication Studies Using Secondary or Nonexperimental Datasets

Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field

Strengthening the foundation of educational psychology by integrating construct validation into open science reform

Recommendations for Replication Research in Special Education

Play It Again, Sam! An Analysis of Constructive Replication in the Organizational Sciences

A glass half full interpretation of the replicability of psychological science

The Reproducibility Movement in Psychology: Does Researcher Gender Affect How People Perceive Scientists With a Failed Replication?

Estimating the reproducibility of psychological science