Over half of psychology studies fail reproducibility test

Monya Baker

DOI: https://doi.org/10.1038/nature.2015.18248

IF: 64.8

2015-08-27

Nature

Abstract:Largest replication study to date casts doubt on many published positive results.

multidisciplinary sciences

What problem does this paper attempt to address?

A glass half full interpretation of the replicability of psychological science

Jeffrey T. Leek,Prasad Patil,Roger D. Peng

DOI: https://doi.org/10.48550/arXiv.1509.08968

2015-09-29

Applications

Abstract:A recent study of the replicability of key psychological findings is a major contribution toward understanding the human side of the scientific process. Despite the careful and nuanced analysis reported in the paper, mass and social media adhered to the simple narrative that only 36% of the studies replicated their original results. Here we show that 77% of the replication effect sizes reported were within a prediction interval based on the original effect size. In this light, the results of Reproducibility Project: Psychology can be viewed as a positive result for the scientific process.
A discipline-wide investigation of the replicability of Psychology papers over the past two decades

Wu Youyou,Yang Yang,Brian Uzzi

DOI: https://doi.org/10.1073/pnas.2208863120

2023-02-01

Abstract:Conjecture about the weak replicability in social sciences has made scholars eager to quantify the scale and scope of replication failure for a discipline. Yet small-scale manual replication methods alone are ill-suited to deal with this big data problem. Here, we conduct a discipline-wide replication census in science. Our sample ( N = 14,126 papers) covers nearly all papers published in the six top-tier Psychology journals over the past 20 y. Using a validated machine learning model that estimates a paper's likelihood of replication, we found evidence that both supports and refutes speculations drawn from a relatively small sample of manual replications. First, we find that a single overall replication rate of Psychology poorly captures the varying degree of replicability among subfields. Second, we find that replication rates are strongly correlated with research methods in all subfields. Experiments replicate at a significantly lower rate than do non-experimental studies. Third, we find that authors' cumulative publication number and citation impact are positively related to the likelihood of replication, while other proxies of research quality and rigor, such as an author's university prestige and a paper's citations, are unrelated to replicability. Finally, contrary to the ideal that media attention should cover replicable research, we find that media attention is positively related to the likelihood of replication failure. Our assessments of the scale and scope of replicability are important next steps toward broadly resolving issues of replicability.

multidisciplinary sciences
Comment on “Estimating the reproducibility of psychological science”

Daniel T Gilbert,Gary King,Stephen Pettigrew,Timothy D Wilson,Daniel T. Gilbert,Timothy D. Wilson

DOI: https://doi.org/10.1126/science.aad7243

IF: 56.9

2016-03-04

Science

Abstract:A paper from the Open Science Collaboration (Research Articles, 28 August 2015, aac4716) attempting to replicate 100 published studies suggests that the reproducibility of psychological science is surprisingly low. We show that this article contains three statistical errors and provides no support for such a conclusion. Indeed, the data are consistent with the opposite conclusion, namely, that the reproducibility of psychological science is quite high.

multidisciplinary sciences
The reproducibility of statistical results in psychological research: An investigation using unpublished raw data.

Richard Artner,Thomas Verliefde,Sara Steegen,Sara Gomes,Frits Traets,Francis Tuerlinckx,Wolf Vanpaemel

DOI: https://doi.org/10.1037/met0000365

IF: 10.929

2021-10-01

Psychological Methods

Abstract:We investigated the reproducibility of the major statistical conclusions drawn in 46 articles published in 2012 in three APA journals. After having identified 232 key statistical claims, we tried to reproduce, for each claim, the test statistic, its degrees of freedom, and the corresponding p value, starting from the raw data that were provided by the authors and closely following the Method section in the article. Out of the 232 claims, we were able to successfully reproduce 163 (70%), 18 of which only by deviating from the article's analytical description. Thirteen (7%) of the 185 claims deemed significant by the authors are no longer so. The reproduction successes were often the result of cumbersome and time-consuming trial-and-error work, suggesting that APA style reporting in conjunction with raw data makes numerical verification at least hard, if not impossible. This article discusses the types of mistakes we could identify and the tediousness of our reproduction efforts in the light of a newly developed taxonomy for reproducibility. We then link our findings with other findings of empirical research on this topic, give practical recommendations on how to achieve reproducibility, and discuss the challenges of large-scale reproducibility checks as well as promising ideas that could considerably increase the reproducibility of psychological research. (PsycInfo Database Record (c) 2021 APA, all rights reserved)

psychology, multidisciplinary
Are Psychology Journals Anti-replication? A Snapshot of Editorial Practices

G. N. Martin,Richard M. Clarke

DOI: https://doi.org/10.3389/fpsyg.2017.00523

IF: 3.8

2017-04-11

Frontiers in Psychology

Abstract:Recent research in psychology has highlighted a number of replication problems in the discipline, with publication bias - the preference for publishing original and positive results, and a resistance to publishing negative results and replications- identified as one reason for replication failure. However, little empirical research exists to demonstrate that journals explicitly refuse to publish replications. We reviewed the instructions to authors and the published aims of 1151 psychology journals and examined whether they indicated that replications were permitted and accepted. We also examined whether journal practices differed across branches of the discipline, and whether editorial practices differed between low and high impact journals. Thirty three journals (3%) stated in their aims or instructions to authors that they accepted replications. There was no difference between high and low impact journals. The implications of these findings for psychology are discussed.

psychology, multidisciplinary
Estimating the reproducibility of psychological science

Open Science Collaboration

DOI: https://doi.org/10.1126/science.aac4716

IF: 56.9

2015-08-28

Science

Abstract:Empirically analyzing empirical evidence One of the central goals in any scientific endeavor is to understand causality. Experiments that seek to demonstrate a cause/effect relation most often manipulate the postulated causal factor. Aarts et al. describe the replication of 100 experiments reported in papers published in 2008 in three high-ranking psychology journals. Assessing whether the replication and the original experiment yielded the same result according to several criteria, they find that about one-third to one-half of the original findings were also observed in the replication study. Science , this issue 10.1126/science.aac4716

multidisciplinary sciences
The Reproducibility Movement in Psychology: Does Researcher Gender Affect How People Perceive Scientists With a Failed Replication?

Leslie Ashburn-Nardo,Corinne A. Moss-Racusin,Jessi L. Smith,Christina M. Sanzari,Theresa K. Vescio,Peter Glick

DOI: https://doi.org/10.3389/fpsyg.2022.823147

IF: 3.8

2022-06-13

Frontiers in Psychology

Abstract:The reproducibility movement in psychology has resulted in numerous highly publicized instances of replication failures. The goal of the present work was to investigate people’s reactions to a psychology replication failure vs. success, and to test whether a failure elicits harsher reactions when the researcher is a woman vs. a man. We examined these questions in a pre-registered experiment with a working adult sample, a conceptual replication of that experiment with a student sample, and an analysis of data compiled and posted by a psychology researcher on their public weblog with the stated goal to improve research replicability by rank-ordering psychology researchers by their “estimated false discovery risk.” Participants in the experiments were randomly assigned to read a news article describing a successful vs. failed replication attempt of original work from a male vs. female psychological scientist, and then completed measures of researcher competence, likability, integrity, perceptions of the research, and behavioral intentions for future interactions with the researcher. In both working adult and student samples, analyses consistently yielded large main effects of replication outcome, but no interaction with researcher gender. Likewise, the coding of weblog data posted in July 2021 indicated that 66.3% of the researchers scrutinized were men and 33.8% were women, and their rank-ordering was not correlated with researcher gender. The lack of support for our pre-registered gender-replication hypothesis is, at first glance, encouraging for women researchers’ careers; however, the substantial effect sizes we observed for replication outcome underscore the tremendous negative impact the reproducibility movement can have on psychologists’ careers. We discuss the implications of such negative perceptions and the possible downstream consequences for women in the field that are essential for future study.

psychology, multidisciplinary
An Excess of Positive Results: Comparing the Standard Psychology Literature With Registered Reports

Anne M. Scheel,Mitchell R. M. J. Schijen,Daniël Lakens

DOI: https://doi.org/10.1177/25152459211007467

2021-04-01

Advances in Methods and Practices in Psychological Science

Abstract:Selectively publishing results that support the tested hypotheses (“positive” results) distorts the available evidence for scientific claims. For the past decade, psychological scientists have been increasingly concerned about the degree of such distortion in their literature. A new publication format has been developed to prevent selective reporting: In Registered Reports (RRs), peer review and the decision to publish take place before results are known. We compared the results in published RRs ( N = 71 as of November 2018) with a random sample of hypothesis-testing studies from the standard literature ( N = 152) in psychology. Analyzing the first hypothesis of each article, we found 96% positive results in standard reports but only 44% positive results in RRs. We discuss possible explanations for this large difference and suggest that a plausible factor is the reduction of publication bias and/or Type I error inflation in the RR literature.
Recent reproducibility estimates indicate that negative evidence is observed over 30 times before publication

Michael Ingre

DOI: https://doi.org/10.48550/arXiv.1605.06414

2016-09-12

Abstract:The Open Science Collaboration recently reported that 36% of published findings from psychological studies were reproducible by independent researchers. We can use this information together with Bayes theorem to estimate the statistical power needed to produce these findings under various assumptions and calculate the expected distribution of positive and negative evidence for a range of prior probabilities of the tested hypotheses; and by comparing this distribution to other findings indicating that >90% of publications in the psychological literature are statistically significant in support of the authors hypothesis, we can estimate the magnitude of publication bias. The results indicate that negative evidence was observed 30--200 times before one was published.

Physics and Society
Participant Nonnaiveté and the reproducibility of cognitive psychology

Rolf A. Zwaan,Diane Pecher,Gabriele Paolacci,Samantha Bouwmeester,Peter Verkoeijen,Katinka Dijkstra,René Zeelenberg

DOI: https://doi.org/10.3758/s13423-017-1348-y

2017-07-25

Abstract:Many argue that there is a reproducibility crisis in psychology. We investigated nine well-known effects from the cognitive psychology literature—three each from the domains of perception/action, memory, and language, respectively—and found that they are highly reproducible. Not only can they be reproduced in online environments, but they also can be reproduced with nonnaïve participants with no reduction of effect size. Apparently, some cognitive tasks are so constraining that they encapsulate behavior from external influences, such as testing situation and prior recent experience with the experiment to yield highly robust effects.

psychology, experimental, mathematical
Replications are informative, particularly when they fail

Maarten Derksen,Jill Morawski

DOI: https://doi.org/10.1177/09593543241269697

2024-10-02

Theory & Psychology

Abstract:Theory &Psychology, Volume 34, Issue 5, Page 597-603, October 2024. Our comment on Archer's enticing article focuses on his extension of Bhaskar's philosophy to psychology, and on direct replication, which the author says is of very limited use in psychology. We only deal with critical realism indirectly. Our arguments boil down to two points: experiments can usefully be seen as performative (and the same goes for research in general), and replication is not problematic in psychology because of the variability of results, but useful for precisely that reason. Replication studies may even inspire psychologists to accept a wider range of epistemic approaches, although there are reasons to be pessimistic about the likelihood of fundamental change in the discipline.

psychology, multidisciplinary
Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015

Colin F. Camerer,Anna Dreber,Felix Holzmeister,Teck-Hua Ho,Jürgen Huber,Magnus Johannesson,Michael Kirchler,Gideon Nave,Brian A. Nosek,Thomas Pfeiffer,Adam Altmejd,Nick Buttrick,Taizan Chan,Yiling Chen,Eskil Forsell,Anup Gampa,Emma Heikensten,Lily Hummer,Taisuke Imai,Siri Isaksson,Dylan Manfredi,Julia Rose,Eric-Jan Wagenmakers,Hang Wu

DOI: https://doi.org/10.1038/s41562-018-0399-z

IF: 24.252

2018-08-27

Nature Human Behaviour

Abstract:Being able to replicate scientific findings is crucial for scientific progress1–15. We replicate 21 systematically selected experimental studies in the social sciences published in Nature and Science between 2010 and 201516–36. The replications follow analysis plans reviewed by the original authors and pre-registered prior to the replications. The replications are high powered, with sample sizes on average about five times higher than in the original studies. We find a significant effect in the same direction as the original study for 13 (62%) studies, and the effect size of the replications is on average about 50% of the original effect size. Replicability varies between 12 (57%) and 14 (67%) studies for complementary replicability indicators. Consistent with these results, the estimated true-positive rate is 67% in a Bayesian analysis. The relative effect size of true positives is estimated to be 71%, suggesting that both false positives and inflated effect sizes of true positives contribute to imperfect reproducibility. Furthermore, we find that peer beliefs of replicability are strongly related to replicability, suggesting that the research community could predict which results would replicate and that failures to replicate were not the result of chance alone.

psychology, experimental,neurosciences,multidisciplinary sciences
Changes in methodological study characteristics in psychology between 2010-2021

Ingmar Böschen

DOI: https://doi.org/10.1371/journal.pone.0283353

IF: 3.7

2023-05-10

PLoS ONE

Abstract:In 2015, the Open Science Collaboration repeated a series of 100 psychological experiments. Since a considerable part of these replications could not confirm the original effects and some of them pointed in the opposite direction, psychological research is said to lack reproducibility. Several general criticisms can explain this finding, such as the standardized use of undirected nil-null hypothesis tests, samples being too small and selective, lack of corrections for multiple testing, but also some widespread questionable research practices and incentives to publish positive results only. A selection of 57,909 articles from 12 renowned journals is processed with the JATSdecoder software to analyze the extent to which several empirical research practices in psychology have changed over the past 12 years. To identify journal- and time-specific changes, the relative use of statistics based on p-values, the number of reported p-values per paper, the relative use of confidence intervals, directed tests, power analysis, Bayesian procedures, non-standard α levels, correction procedures for multiple testing, and median sample sizes are analyzed for articles published between 2010 and 2015 and after 2015, and in more detail for every included journal and year of publication. In addition, the origin of authorships is analyzed over time. Compared to articles that were published in and before 2015, the median number of reported p-values per article has decreased from 14 to 12, whereas the median proportion of significant p-values per article remained constant at 69%. While reports of effect sizes and confidence intervals have increased, the α level is usually set to the default value of .05. The use of corrections for multiple testing has decreased. Although uncommon in each case (4% in total), directed testing is used less frequently, while Bayesian inference has become more common after 2015. The overall median estimated sample size has increased from 105 to 190.
Replication, falsification, and the crisis of confidence in social psychology

Brian D. Earp,David Trafimow

DOI: https://doi.org/10.3389/fpsyg.2015.00621

IF: 3.8

2015-05-19

Frontiers in Psychology

Abstract:The (latest) crisis in confidence in social psychology has generated much heated discussion about the importance of replication, including how it should be carried out as well as interpreted by scholars in the field. For example, what does it mean if a replication attempt "fails"-does it mean that the original results, or the theory that predicted them, have been falsified? And how should "failed" replications affect our belief in the validity of the original research? In this paper, we consider the replication debate from a historical and philosophical perspective, and provide a conceptual analysis of both replication and falsification as they pertain to this important discussion. Along the way, we highlight the importance of auxiliary assumptions (for both testing theories and attempting replications), and introduce a Bayesian framework for assessing "failed" replications in terms of how they should affect our confidence in original findings.

psychology, multidisciplinary
A critical scientific evaluation of a purportedly negative data report – response to Seneviratne et al. 2022

Bankole Johnson,Giovanni Addolorato,Otto Lesch,Lei Liu,Zachary A. Rodd

DOI: https://doi.org/10.3389/fpsyt.2023.1271229

IF: 4.7

2023-10-05

Frontiers in Psychiatry

Abstract:A core principle in the pursuit of scientific knowledge is that science is self-correcting and that important results should be replicable. Hypotheses need to be reinforced, adjusted, or rejected when novel results are obtained. Replication of results confirms hypotheses and enhances their integration into scientific practice. In contrast, publication of substantiated and replicated negative findings (i.e., non-significant or opposite findings) can be the basis to reject erroneous hypotheses or develop alternative strategies for investigation. Replication is a problem in all research fields. The Psychology Reproductivity Project reported that only 36% of 'highly influential' published research in highly ranked journals were reproduced. Similar to positive data, negative data can be flawed. Errors in a negative data set can be based on methodology, statistics, conceptual defects, and flawed peer review. The peer review process has received progressive scrutiny. A large-scale review of the peer review process of manuscripts submitted to the British Medical Journal group indicated that the process could be characterized as inconsistent, inaccurate, and biased. Further analysis indicated that the peer process is easily manipulated, indicative of a failed system, is a major factor behind the lack of replication in science (acceptance of flawed manuscripts), suppresses opposing scientific evidence and views, and causes gaps in and lack of growth of science. Complicating the integrity of scientific publication is the role of Editors/Researchers. Ethical guidelines exist for major publishing houses about editorial ethics, behavior, and practice.

psychiatry
Why Most Research Findings About Psi Are False: The Replicability Crisis, the Psi Paradox and the Myth of Sisyphus

Thomas Rabeyron

DOI: https://doi.org/10.3389/fpsyg.2020.562992

2020-09-18

Abstract:The replicability crisis in psychology has been influenced by the results of nine experiments conducted by Bem (2011) and presented as supporting the existence of precognition. In this paper, we hope to show how the debate concerning these experiments could be an opportunity to develop original thinking about psychology and replicability. After a few preliminary remarks about psi and scientific epistemology, we examine how psi results lead to a paradox which questions how appropriate the scientific method is to psi research. This paradox highlights a problem in the way experiments are conducted in psi research and its potential consequence on mainstream research in psychology. Two classical experiments - the Ganzfeld protocol and the Bem studies - are then analyzed in order to illustrate this paradox and its consequences. Mainstream research is also addressed in the broader context of the replication crisis, decline effect and questionable research practices. Several perspectives for future research are proposed in conclusion and underline the heuristic value of psi studies for psychology.
Addressing the “Replication Crisis”: Using Original Studies to Design Replication Studies with Appropriate Statistical Power

Samantha F. Anderson,Scott E. Maxwell

DOI: https://doi.org/10.1080/00273171.2017.1289361

2017-03-07

Multivariate Behavioral Research

Abstract:Psychology is undergoing a replication crisis. The discussion surrounding this crisis has centered on mistrust of previous findings. Researchers planning replication studies often use the original study sample effect size as the basis for sample size planning. However, this strategy ignores uncertainty and publication bias in estimated effect sizes, resulting in overly optimistic calculations. A psychologist who intends to obtain power of .80 in the replication study, and performs calculations accordingly, may have an actual power lower than .80. We performed simulations to reveal the magnitude of the difference between actual and intended power based on common sample size planning strategies and assessed the performance of methods that aim to correct for effect size uncertainty and/or bias. Our results imply that even if original studies reflect actual phenomena and were conducted in the absence of questionable research practices, popular approaches to designing replication studies may result in a low success rate, especially if the original study is underpowered. Methods correcting for bias and/or uncertainty generally had higher actual power, but were not a panacea for an underpowered original study. Thus, it becomes imperative that 1) original studies are adequately powered and 2) replication studies are designed with methods that are more likely to yield the intended level of power.

psychology, experimental,mathematics, interdisciplinary applications,statistics & probability,social sciences, mathematical methods
The state of play of reproducibility in Statistics: an empirical analysis

Xin Xiong,Ivor Cribben

DOI: https://doi.org/10.48550/arXiv.2209.15602

2022-10-01

Abstract:Reproducibility, the ability to reproduce the results of published papers or studies using their computer code and data, is a cornerstone of reliable scientific methodology. Studies where results cannot be reproduced by the scientific community should be treated with caution. Over the past decade, the importance of reproducible research has been frequently stressed in a wide range of scientific journals such as \textit{Nature} and \textit{Science} and international magazines such as \textit{The Economist}. However, multiple studies have demonstrated that scientific results are often not reproducible across research areas such as psychology and medicine. Statistics, the science concerned with developing and studying methods for collecting, analyzing, interpreting and presenting empirical data, prides itself on its openness when it comes to sharing both computer code and data. In this paper, we examine reproducibility in the field of statistics by attempting to reproduce the results in 93 published papers in prominent journals utilizing functional magnetic resonance imaging (fMRI) data during the 2010-2021 period. Overall, from both the computer code and the data perspective, among all the 93 examined papers, we could only reproduce the results in 14 (15.1%) papers, that is, the papers provide both executable computer code (or software) with the real fMRI data, and our results matched the results in the paper. Finally, we conclude with some author-specific and journal-specific recommendations to improve the research reproducibility in statistics.

Applications
An illustration of reproducibility in neuroscience research in the absence of selective reporting

Xiang-Zhen Kong,Clyde Francks,

DOI: https://doi.org/10.1101/866301

2019-12-06

Abstract:Abstract The problem of poor reproducibility of scientific findings has received much attention over recent years, in a variety of fields including psychology and neuroscience. The problem has been partly attributed to publication bias and unwanted practices such as p -hacking. Low statistical power in individual studies is also understood to be an important factor. In a recent multi-site collaborative study, we mapped brain anatomical left-right asymmetries for regional measures of surface area and cortical thickness, in 99 MRI datasets from around the world, for a total of over 17,000 participants. In the present study, we re-visited these hemispheric effects from the perspective of reproducibility. Within each dataset, we considered that an effect had been reproduced when it matched the meta-analytic effect from the 98 other datasets, in terms of effect direction and uncorrected significance at p <0.05. In this sense, the results within each dataset were viewed as coming from separate studies in an ‘ideal publishing environment’, i.e. free from selective reporting and p hacking. We found an average reproducibility rate per dataset, over all effects, of 63.2% ( SD = 22.9%, min = 22.2%, max = 97.0%). As expected, reproducibility was higher for larger effects and in larger datasets. There is clearly substantial room to improve reproducibility in brain MRI research through increasing statistical power. These findings constitute an empirical illustration of reproducibility in the absence of publication bias or p hacking, when assessing realistic biological effects in heterogeneous neuroscience data, and given typically-used sample sizes.
The “replication crisis” and trust in psychological science: How reforms shape public trust in psychology

Nicole Methner,Barbara Dahme,Claudia Menzel

DOI: https://doi.org/10.32872/spb.9665

2023-11-17

Social Psychological Bulletin

Abstract:Failed replications can jeopardize public trust in psychological science and recent findings cast doubt on the idea that self-corrections and reforms can rebuild this trust. These findings are in contrast to trust repair research that proposes changes in transparency, norms, and policies as trust repair mechanisms. This raises the question of whether the used experimental material is one reason behind these unexpected findings. Previous studies used short texts that may give too little information on the replication crisis and initiated reforms in the field. In a pre-registered experiment (N = 390), we, therefore, tested whether comprehensive information about the replication crisis and reforms increases public trust in psychology, compared to a control condition that only informs about the replication crisis. To give comprehensive information, we created an animated video for each experimental condition. After watching the video, participants indicated their trust in researchers, trust in past research findings, and trust in current research findings. As expected and in line with trust repair research, information about reforms increased trust in researchers and in current (vs. past) research, compared with information about the replication crisis and its causes only. We discuss the generalizability of our results and implications for communicating the replication crisis to the public.

Over half of psychology studies fail reproducibility test

A glass half full interpretation of the replicability of psychological science

A discipline-wide investigation of the replicability of Psychology papers over the past two decades

Comment on “Estimating the reproducibility of psychological science”

The reproducibility of statistical results in psychological research: An investigation using unpublished raw data.

Are Psychology Journals Anti-replication? A Snapshot of Editorial Practices

Estimating the reproducibility of psychological science

The Reproducibility Movement in Psychology: Does Researcher Gender Affect How People Perceive Scientists With a Failed Replication?

An Excess of Positive Results: Comparing the Standard Psychology Literature With Registered Reports

Recent reproducibility estimates indicate that negative evidence is observed over 30 times before publication

Participant Nonnaiveté and the reproducibility of cognitive psychology

Replications are informative, particularly when they fail

Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015

Changes in methodological study characteristics in psychology between 2010-2021

Replication, falsification, and the crisis of confidence in social psychology

A critical scientific evaluation of a purportedly negative data report – response to Seneviratne et al. 2022

Why Most Research Findings About Psi Are False: The Replicability Crisis, the Psi Paradox and the Myth of Sisyphus

Addressing the “Replication Crisis”: Using Original Studies to Design Replication Studies with Appropriate Statistical Power

The state of play of reproducibility in Statistics: an empirical analysis

An illustration of reproducibility in neuroscience research in the absence of selective reporting

The “replication crisis” and trust in psychological science: How reforms shape public trust in psychology