Psychology’s reproducibility problem is exaggerated – say psychologists

Monya Baker

DOI: https://doi.org/10.1038/nature.2016.19498

IF: 64.8

2016-03-03

Nature

Abstract:Reanalysis of last year's enormous replication study argues that there is no need to be so pessimistic.

multidisciplinary sciences

What problem does this paper attempt to address?

Comment on “Estimating the reproducibility of psychological science”

Daniel T Gilbert,Gary King,Stephen Pettigrew,Timothy D Wilson,Daniel T. Gilbert,Timothy D. Wilson

DOI: https://doi.org/10.1126/science.aad7243

IF: 56.9

2016-03-04

Science

Abstract:A paper from the Open Science Collaboration (Research Articles, 28 August 2015, aac4716) attempting to replicate 100 published studies suggests that the reproducibility of psychological science is surprisingly low. We show that this article contains three statistical errors and provides no support for such a conclusion. Indeed, the data are consistent with the opposite conclusion, namely, that the reproducibility of psychological science is quite high.

multidisciplinary sciences
A discipline-wide investigation of the replicability of Psychology papers over the past two decades

Wu Youyou,Yang Yang,Brian Uzzi

DOI: https://doi.org/10.1073/pnas.2208863120

2023-02-01

Abstract:Conjecture about the weak replicability in social sciences has made scholars eager to quantify the scale and scope of replication failure for a discipline. Yet small-scale manual replication methods alone are ill-suited to deal with this big data problem. Here, we conduct a discipline-wide replication census in science. Our sample ( N = 14,126 papers) covers nearly all papers published in the six top-tier Psychology journals over the past 20 y. Using a validated machine learning model that estimates a paper's likelihood of replication, we found evidence that both supports and refutes speculations drawn from a relatively small sample of manual replications. First, we find that a single overall replication rate of Psychology poorly captures the varying degree of replicability among subfields. Second, we find that replication rates are strongly correlated with research methods in all subfields. Experiments replicate at a significantly lower rate than do non-experimental studies. Third, we find that authors' cumulative publication number and citation impact are positively related to the likelihood of replication, while other proxies of research quality and rigor, such as an author's university prestige and a paper's citations, are unrelated to replicability. Finally, contrary to the ideal that media attention should cover replicable research, we find that media attention is positively related to the likelihood of replication failure. Our assessments of the scale and scope of replicability are important next steps toward broadly resolving issues of replicability.

multidisciplinary sciences
A glass half full interpretation of the replicability of psychological science

Jeffrey T. Leek,Prasad Patil,Roger D. Peng

DOI: https://doi.org/10.48550/arXiv.1509.08968

2015-09-29

Applications

Abstract:A recent study of the replicability of key psychological findings is a major contribution toward understanding the human side of the scientific process. Despite the careful and nuanced analysis reported in the paper, mass and social media adhered to the simple narrative that only 36% of the studies replicated their original results. Here we show that 77% of the replication effect sizes reported were within a prediction interval based on the original effect size. In this light, the results of Reproducibility Project: Psychology can be viewed as a positive result for the scientific process.
Replications are informative, particularly when they fail

Maarten Derksen,Jill Morawski

DOI: https://doi.org/10.1177/09593543241269697

2024-10-02

Theory & Psychology

Abstract:Theory &Psychology, Volume 34, Issue 5, Page 597-603, October 2024. Our comment on Archer's enticing article focuses on his extension of Bhaskar's philosophy to psychology, and on direct replication, which the author says is of very limited use in psychology. We only deal with critical realism indirectly. Our arguments boil down to two points: experiments can usefully be seen as performative (and the same goes for research in general), and replication is not problematic in psychology because of the variability of results, but useful for precisely that reason. Replication studies may even inspire psychologists to accept a wider range of epistemic approaches, although there are reasons to be pessimistic about the likelihood of fundamental change in the discipline.

psychology, multidisciplinary
The reproducibility of statistical results in psychological research: An investigation using unpublished raw data.

Richard Artner,Thomas Verliefde,Sara Steegen,Sara Gomes,Frits Traets,Francis Tuerlinckx,Wolf Vanpaemel

DOI: https://doi.org/10.1037/met0000365

IF: 10.929

2021-10-01

Psychological Methods

Abstract:We investigated the reproducibility of the major statistical conclusions drawn in 46 articles published in 2012 in three APA journals. After having identified 232 key statistical claims, we tried to reproduce, for each claim, the test statistic, its degrees of freedom, and the corresponding p value, starting from the raw data that were provided by the authors and closely following the Method section in the article. Out of the 232 claims, we were able to successfully reproduce 163 (70%), 18 of which only by deviating from the article's analytical description. Thirteen (7%) of the 185 claims deemed significant by the authors are no longer so. The reproduction successes were often the result of cumbersome and time-consuming trial-and-error work, suggesting that APA style reporting in conjunction with raw data makes numerical verification at least hard, if not impossible. This article discusses the types of mistakes we could identify and the tediousness of our reproduction efforts in the light of a newly developed taxonomy for reproducibility. We then link our findings with other findings of empirical research on this topic, give practical recommendations on how to achieve reproducibility, and discuss the challenges of large-scale reproducibility checks as well as promising ideas that could considerably increase the reproducibility of psychological research. (PsycInfo Database Record (c) 2021 APA, all rights reserved)

psychology, multidisciplinary
Estimating the reproducibility of psychological science

Open Science Collaboration

DOI: https://doi.org/10.1126/science.aac4716

IF: 56.9

2015-08-28

Science

Abstract:Empirically analyzing empirical evidence One of the central goals in any scientific endeavor is to understand causality. Experiments that seek to demonstrate a cause/effect relation most often manipulate the postulated causal factor. Aarts et al. describe the replication of 100 experiments reported in papers published in 2008 in three high-ranking psychology journals. Assessing whether the replication and the original experiment yielded the same result according to several criteria, they find that about one-third to one-half of the original findings were also observed in the replication study. Science , this issue 10.1126/science.aac4716

multidisciplinary sciences
Participant Nonnaiveté and the reproducibility of cognitive psychology

Rolf A. Zwaan,Diane Pecher,Gabriele Paolacci,Samantha Bouwmeester,Peter Verkoeijen,Katinka Dijkstra,René Zeelenberg

DOI: https://doi.org/10.3758/s13423-017-1348-y

2017-07-25

Abstract:Many argue that there is a reproducibility crisis in psychology. We investigated nine well-known effects from the cognitive psychology literature—three each from the domains of perception/action, memory, and language, respectively—and found that they are highly reproducible. Not only can they be reproduced in online environments, but they also can be reproduced with nonnaïve participants with no reduction of effect size. Apparently, some cognitive tasks are so constraining that they encapsulate behavior from external influences, such as testing situation and prior recent experience with the experiment to yield highly robust effects.

psychology, experimental, mathematical
Are Psychology Journals Anti-replication? A Snapshot of Editorial Practices

G. N. Martin,Richard M. Clarke

DOI: https://doi.org/10.3389/fpsyg.2017.00523

IF: 3.8

2017-04-11

Frontiers in Psychology

Abstract:Recent research in psychology has highlighted a number of replication problems in the discipline, with publication bias - the preference for publishing original and positive results, and a resistance to publishing negative results and replications- identified as one reason for replication failure. However, little empirical research exists to demonstrate that journals explicitly refuse to publish replications. We reviewed the instructions to authors and the published aims of 1151 psychology journals and examined whether they indicated that replications were permitted and accepted. We also examined whether journal practices differed across branches of the discipline, and whether editorial practices differed between low and high impact journals. Thirty three journals (3%) stated in their aims or instructions to authors that they accepted replications. There was no difference between high and low impact journals. The implications of these findings for psychology are discussed.

psychology, multidisciplinary
An Excess of Positive Results: Comparing the Standard Psychology Literature With Registered Reports

Anne M. Scheel,Mitchell R. M. J. Schijen,Daniël Lakens

DOI: https://doi.org/10.1177/25152459211007467

2021-04-01

Advances in Methods and Practices in Psychological Science

Abstract:Selectively publishing results that support the tested hypotheses (“positive” results) distorts the available evidence for scientific claims. For the past decade, psychological scientists have been increasingly concerned about the degree of such distortion in their literature. A new publication format has been developed to prevent selective reporting: In Registered Reports (RRs), peer review and the decision to publish take place before results are known. We compared the results in published RRs ( N = 71 as of November 2018) with a random sample of hypothesis-testing studies from the standard literature ( N = 152) in psychology. Analyzing the first hypothesis of each article, we found 96% positive results in standard reports but only 44% positive results in RRs. We discuss possible explanations for this large difference and suggest that a plausible factor is the reduction of publication bias and/or Type I error inflation in the RR literature.
The Reproducibility Movement in Psychology: Does Researcher Gender Affect How People Perceive Scientists With a Failed Replication?

Leslie Ashburn-Nardo,Corinne A. Moss-Racusin,Jessi L. Smith,Christina M. Sanzari,Theresa K. Vescio,Peter Glick

DOI: https://doi.org/10.3389/fpsyg.2022.823147

IF: 3.8

2022-06-13

Frontiers in Psychology

Abstract:The reproducibility movement in psychology has resulted in numerous highly publicized instances of replication failures. The goal of the present work was to investigate people’s reactions to a psychology replication failure vs. success, and to test whether a failure elicits harsher reactions when the researcher is a woman vs. a man. We examined these questions in a pre-registered experiment with a working adult sample, a conceptual replication of that experiment with a student sample, and an analysis of data compiled and posted by a psychology researcher on their public weblog with the stated goal to improve research replicability by rank-ordering psychology researchers by their “estimated false discovery risk.” Participants in the experiments were randomly assigned to read a news article describing a successful vs. failed replication attempt of original work from a male vs. female psychological scientist, and then completed measures of researcher competence, likability, integrity, perceptions of the research, and behavioral intentions for future interactions with the researcher. In both working adult and student samples, analyses consistently yielded large main effects of replication outcome, but no interaction with researcher gender. Likewise, the coding of weblog data posted in July 2021 indicated that 66.3% of the researchers scrutinized were men and 33.8% were women, and their rank-ordering was not correlated with researcher gender. The lack of support for our pre-registered gender-replication hypothesis is, at first glance, encouraging for women researchers’ careers; however, the substantial effect sizes we observed for replication outcome underscore the tremendous negative impact the reproducibility movement can have on psychologists’ careers. We discuss the implications of such negative perceptions and the possible downstream consequences for women in the field that are essential for future study.

psychology, multidisciplinary
Replicating is difficult but necessary, and methodology can help

David Trafimow

DOI: https://doi.org/10.1177/09593543241265912

2024-10-02

Theory & Psychology

Abstract:Theory &Psychology, Volume 34, Issue 5, Page 591-596, October 2024. Archer advised psychology researchers to reject replication and experimentalism. The present comment counters Archer's recommendations. Researchers should be more, not less, concerned with methodology to improve the quality of replication attempts and more convincingly test theories.

psychology, multidisciplinary
Estimating the Prevalence of Transparency and Reproducibility-Related Research Practices in Psychology (2014–2017)

Tom E. Hardwicke,Robert T. Thibault,Jessica E. Kosie,Joshua D. Wallach,Mallory C. Kidwell,John P. A. Ioannidis

DOI: https://doi.org/10.1177/1745691620979806

2021-03-08

Perspectives on Psychological Science

Abstract:Psychologists are navigating an unprecedented period of introspection about the credibility and utility of their discipline. Reform initiatives emphasize the benefits of transparency and reproducibility-related research practices; however, adoption across the psychology literature is unknown. Estimating the prevalence of such practices will help to gauge the collective impact of reform initiatives, track progress over time, and calibrate future efforts. To this end, we manually examined a random sample of 250 psychology articles published between 2014 and 2017. Over half of the articles were publicly available (154/237, 65%, 95% confidence interval [CI] = [59%, 71%]); however, sharing of research materials (26/183; 14%, 95% CI = [10%, 19%]), study protocols (0/188; 0%, 95% CI = [0%, 1%]), raw data (4/188; 2%, 95% CI = [1%, 4%]), and analysis scripts (1/188; 1%, 95% CI = [0%, 1%]) was rare. Preregistration was also uncommon (5/188; 3%, 95% CI = [1%, 5%]). Many articles included a funding disclosure statement (142/228; 62%, 95% CI = [56%, 69%]), but conflict-of-interest statements were less common (88/228; 39%, 95% CI = [32%, 45%]). Replication studies were rare (10/188; 5%, 95% CI = [3%, 8%]), and few studies were included in systematic reviews (21/183; 11%, 95% CI = [8%, 16%]) or meta-analyses (12/183; 7%, 95% CI = [4%, 10%]). Overall, the results suggest that transparency and reproducibility-related research practices were far from routine. These findings establish baseline prevalence estimates against which future progress toward increasing the credibility and utility of psychology research can be compared.

psychology, multidisciplinary
Perspectives on Machine Learning from Psychology's Reproducibility Crisis

Samuel J. Bell,Onno P. Kampman

DOI: https://doi.org/10.48550/arXiv.2104.08878

2021-04-23

Abstract:In the early 2010s, a crisis of reproducibility rocked the field of psychology. Following a period of reflection, the field has responded with radical reform of its scientific practices. More recently, similar questions about the reproducibility of machine learning research have also come to the fore. In this short paper, we present select ideas from psychology's reformation, translating them into relevance for a machine learning audience.

Machine Learning,Artificial Intelligence
Addressing the “Replication Crisis”: Using Original Studies to Design Replication Studies with Appropriate Statistical Power

Samantha F. Anderson,Scott E. Maxwell

DOI: https://doi.org/10.1080/00273171.2017.1289361

2017-03-07

Multivariate Behavioral Research

Abstract:Psychology is undergoing a replication crisis. The discussion surrounding this crisis has centered on mistrust of previous findings. Researchers planning replication studies often use the original study sample effect size as the basis for sample size planning. However, this strategy ignores uncertainty and publication bias in estimated effect sizes, resulting in overly optimistic calculations. A psychologist who intends to obtain power of .80 in the replication study, and performs calculations accordingly, may have an actual power lower than .80. We performed simulations to reveal the magnitude of the difference between actual and intended power based on common sample size planning strategies and assessed the performance of methods that aim to correct for effect size uncertainty and/or bias. Our results imply that even if original studies reflect actual phenomena and were conducted in the absence of questionable research practices, popular approaches to designing replication studies may result in a low success rate, especially if the original study is underpowered. Methods correcting for bias and/or uncertainty generally had higher actual power, but were not a panacea for an underpowered original study. Thus, it becomes imperative that 1) original studies are adequately powered and 2) replication studies are designed with methods that are more likely to yield the intended level of power.

psychology, experimental,mathematics, interdisciplinary applications,statistics & probability,social sciences, mathematical methods
Why Most Research Findings About Psi Are False: The Replicability Crisis, the Psi Paradox and the Myth of Sisyphus

Thomas Rabeyron

DOI: https://doi.org/10.3389/fpsyg.2020.562992

2020-09-18

Abstract:The replicability crisis in psychology has been influenced by the results of nine experiments conducted by Bem (2011) and presented as supporting the existence of precognition. In this paper, we hope to show how the debate concerning these experiments could be an opportunity to develop original thinking about psychology and replicability. After a few preliminary remarks about psi and scientific epistemology, we examine how psi results lead to a paradox which questions how appropriate the scientific method is to psi research. This paradox highlights a problem in the way experiments are conducted in psi research and its potential consequence on mainstream research in psychology. Two classical experiments - the Ganzfeld protocol and the Bem studies - are then analyzed in order to illustrate this paradox and its consequences. Mainstream research is also addressed in the broader context of the replication crisis, decline effect and questionable research practices. Several perspectives for future research are proposed in conclusion and underline the heuristic value of psi studies for psychology.
Are replications mainstream now? A comparison of support for replications expressed in the policies of social psychology journals in 2015 and 2022

Ann-Kathrin Torka,Jens Mazei,Joachim Hüffmeier

DOI: https://doi.org/10.32872/spb.9695

2023-11-17

Social Psychological Bulletin

Abstract:A decade ago, replications were typically not conducted and appreciated in social psychology, although replications play a central role in ensuring trust in scientific fields. Without systematic replication efforts, it is not clear whether findings are trustworthy. As journals can function as gatekeepers for publications, they can influence whether researchers conduct (and publish) replications. Yet, the scholarly culture in social psychology might have changed over the last decade because numerous highly visible studies did not replicate past findings. In light of these insights and the resulting learning opportunities for the field, we predicted an increase in the expressed support for replications in the policies of social psychology journals from 2015 (i.e., the year the replication problem became widely known) to 2022. We coded whether and how replications were mentioned in the author guidelines on the websites of social psychology journals (N = 51). As expected, replications were welcomed more often in 2022 (25%) than they were in 2015 (12%), but they were not mentioned on the websites of most journals (71% in 2022 vs. 82% in 2015). An exploratory analysis suggested that journals that expressed support for replications on their websites were also more likely to publish articles about replication. Further, exploratory analyses of the journals’ TOP factors indicated similar rates of support for replications as for other rigor and transparency promoting policies. In sum, our findings suggest that appreciation for replication has increased, but is not yet part of mainstream culture in social psychology.
Changes in methodological study characteristics in psychology between 2010-2021

Ingmar Böschen

DOI: https://doi.org/10.1371/journal.pone.0283353

IF: 3.7

2023-05-10

PLoS ONE

Abstract:In 2015, the Open Science Collaboration repeated a series of 100 psychological experiments. Since a considerable part of these replications could not confirm the original effects and some of them pointed in the opposite direction, psychological research is said to lack reproducibility. Several general criticisms can explain this finding, such as the standardized use of undirected nil-null hypothesis tests, samples being too small and selective, lack of corrections for multiple testing, but also some widespread questionable research practices and incentives to publish positive results only. A selection of 57,909 articles from 12 renowned journals is processed with the JATSdecoder software to analyze the extent to which several empirical research practices in psychology have changed over the past 12 years. To identify journal- and time-specific changes, the relative use of statistics based on p-values, the number of reported p-values per paper, the relative use of confidence intervals, directed tests, power analysis, Bayesian procedures, non-standard α levels, correction procedures for multiple testing, and median sample sizes are analyzed for articles published between 2010 and 2015 and after 2015, and in more detail for every included journal and year of publication. In addition, the origin of authorships is analyzed over time. Compared to articles that were published in and before 2015, the median number of reported p-values per article has decreased from 14 to 12, whereas the median proportion of significant p-values per article remained constant at 69%. While reports of effect sizes and confidence intervals have increased, the α level is usually set to the default value of .05. The use of corrections for multiple testing has decreased. Although uncommon in each case (4% in total), directed testing is used less frequently, while Bayesian inference has become more common after 2015. The overall median estimated sample size has increased from 105 to 190.
Replication, falsification, and the crisis of confidence in social psychology

Brian D. Earp,David Trafimow

DOI: https://doi.org/10.3389/fpsyg.2015.00621

IF: 3.8

2015-05-19

Frontiers in Psychology

Abstract:The (latest) crisis in confidence in social psychology has generated much heated discussion about the importance of replication, including how it should be carried out as well as interpreted by scholars in the field. For example, what does it mean if a replication attempt "fails"-does it mean that the original results, or the theory that predicted them, have been falsified? And how should "failed" replications affect our belief in the validity of the original research? In this paper, we consider the replication debate from a historical and philosophical perspective, and provide a conceptual analysis of both replication and falsification as they pertain to this important discussion. Along the way, we highlight the importance of auxiliary assumptions (for both testing theories and attempting replications), and introduce a Bayesian framework for assessing "failed" replications in terms of how they should affect our confidence in original findings.

psychology, multidisciplinary
The “replication crisis” and trust in psychological science: How reforms shape public trust in psychology

Nicole Methner,Barbara Dahme,Claudia Menzel

DOI: https://doi.org/10.32872/spb.9665

2023-11-17

Social Psychological Bulletin

Abstract:Failed replications can jeopardize public trust in psychological science and recent findings cast doubt on the idea that self-corrections and reforms can rebuild this trust. These findings are in contrast to trust repair research that proposes changes in transparency, norms, and policies as trust repair mechanisms. This raises the question of whether the used experimental material is one reason behind these unexpected findings. Previous studies used short texts that may give too little information on the replication crisis and initiated reforms in the field. In a pre-registered experiment (N = 390), we, therefore, tested whether comprehensive information about the replication crisis and reforms increases public trust in psychology, compared to a control condition that only informs about the replication crisis. To give comprehensive information, we created an animated video for each experimental condition. After watching the video, participants indicated their trust in researchers, trust in past research findings, and trust in current research findings. As expected and in line with trust repair research, information about reforms increased trust in researchers and in current (vs. past) research, compared with information about the replication crisis and its causes only. We discuss the generalizability of our results and implications for communicating the replication crisis to the public.
The state of play of reproducibility in Statistics: an empirical analysis

Xin Xiong,Ivor Cribben

DOI: https://doi.org/10.48550/arXiv.2209.15602

2022-10-01

Abstract:Reproducibility, the ability to reproduce the results of published papers or studies using their computer code and data, is a cornerstone of reliable scientific methodology. Studies where results cannot be reproduced by the scientific community should be treated with caution. Over the past decade, the importance of reproducible research has been frequently stressed in a wide range of scientific journals such as \textit{Nature} and \textit{Science} and international magazines such as \textit{The Economist}. However, multiple studies have demonstrated that scientific results are often not reproducible across research areas such as psychology and medicine. Statistics, the science concerned with developing and studying methods for collecting, analyzing, interpreting and presenting empirical data, prides itself on its openness when it comes to sharing both computer code and data. In this paper, we examine reproducibility in the field of statistics by attempting to reproduce the results in 93 published papers in prominent journals utilizing functional magnetic resonance imaging (fMRI) data during the 2010-2021 period. Overall, from both the computer code and the data perspective, among all the 93 examined papers, we could only reproduce the results in 14 (15.1%) papers, that is, the papers provide both executable computer code (or software) with the real fMRI data, and our results matched the results in the paper. Finally, we conclude with some author-specific and journal-specific recommendations to improve the research reproducibility in statistics.

Applications

Psychology’s reproducibility problem is exaggerated – say psychologists

Comment on “Estimating the reproducibility of psychological science”

A discipline-wide investigation of the replicability of Psychology papers over the past two decades

A glass half full interpretation of the replicability of psychological science

Replications are informative, particularly when they fail

The reproducibility of statistical results in psychological research: An investigation using unpublished raw data.

Estimating the reproducibility of psychological science

Participant Nonnaiveté and the reproducibility of cognitive psychology

Are Psychology Journals Anti-replication? A Snapshot of Editorial Practices

An Excess of Positive Results: Comparing the Standard Psychology Literature With Registered Reports

The Reproducibility Movement in Psychology: Does Researcher Gender Affect How People Perceive Scientists With a Failed Replication?

Replicating is difficult but necessary, and methodology can help

Estimating the Prevalence of Transparency and Reproducibility-Related Research Practices in Psychology (2014–2017)

Perspectives on Machine Learning from Psychology's Reproducibility Crisis

Addressing the “Replication Crisis”: Using Original Studies to Design Replication Studies with Appropriate Statistical Power

Why Most Research Findings About Psi Are False: The Replicability Crisis, the Psi Paradox and the Myth of Sisyphus

Are replications mainstream now? A comparison of support for replications expressed in the policies of social psychology journals in 2015 and 2022

Changes in methodological study characteristics in psychology between 2010-2021

Replication, falsification, and the crisis of confidence in social psychology

The “replication crisis” and trust in psychological science: How reforms shape public trust in psychology

The state of play of reproducibility in Statistics: an empirical analysis