Publish peer reviews

Jessica K. Polka,Robert Kiley,Boyana Konforti,Bodo Stern,Ronald D. Vale

DOI: https://doi.org/10.1038/d41586-018-06032-w

IF: 64.8

2018-08-01

Nature

Abstract:Jessica K. Polka and colleagues call on journals to sign a pledge to make reviewers’ anonymous comments part of the official scientific record.

multidisciplinary sciences

What problem does this paper attempt to address?

Motivating and Maintaining Ethics, Equity, Effectiveness, Efficiency, and Expertise in Peer Review

Adam Craig,Christina Lee,Nithyaa Bala,Carl Taswell,

DOI: https://doi.org/10.48085/i5b147d9d

2022-06-30

Brainiacs Journal of Brain Imaging And Computing Sciences

Abstract:Scientists who engage in science and the scientific endeavor should seek truth with conviction of morals and commitment to ethics. While the number of publications continues to increase, the number of retractions has increased at a faster rate. Journals publish fraudulent research papers despite claims of peer review and adherence to publishing ethics. Nevertheless, appropriate ethical peer review will remain a gatekeeper when selecting research manuscripts in scholarly publishing and approving research applications for grant funding. However, this peer review must become more open, fair, transparent, equitable, and just with new recommendations and guidelines for reproducible and accountable reviews that support and promote fair citation and citational justice. We should engineer this new peer-review process with modern informatics technology and information science to provide and defend better safeguards for truth and integrity, to clarify and maintain the provenance of information and ideas, and to rebuild and restore trust in scholarly research institutions. Indeed, this new approach will be necessary in the current post-truth era to counter the ease and speed with which mis-information, dis-information, anti-information, caco-information, and mal-information spread through the internet, web, news, and social media. The most important question for application of new peer-review methods to these information wars should be ‘Who does what when?’ in support of reproducible and accountable reviews. Who refers to the authors, reviewers, editors, and publishers as participants in the review process. What refers to disclosure of the participants' identities, the material content of author manuscripts and reviewer commentaries, and other communications between authors and reviewers. When refers to tracking the sequential points in time for which disclosure of whose identity, which content, and which communication at which step of the peer-review process for which audience of readers and reviewers. We believe that quality peer review, and peer review of peer review, must be motivated and maintained by elevating their status and prestige to an art and a science. Both peer review itself and peer review analyses of peer reviews should be incentivised by publishing peer reviews as citable references separately from the research report reviewed while crossreferenced and crosslinked to the report reviewed.
Decentralized Peer Review in Open Science: A Mechanism Proposal

Andreas Finke,Thomas Hensel

2024-04-28

Abstract:Peer review is a laborious, yet essential, part of academic publishing with crucial impact on the scientific endeavor. The current lack of incentives and transparency harms the credibility of this process. Researchers are neither rewarded for superior nor penalized for bad reviews. Additionally, confidential reports cause a loss of insights and make the review process vulnerable to scientific misconduct. We propose a community-owned and -governed system that 1) remunerates reviewers for their efforts, 2) publishes the (anonymized) reports for scrutiny by the community, 3) tracks reputation of reviewers and 4) provides digital certificates. Automated by transparent smart-contract blockchain technology, the system aims to increase quality and speed of peer review while lowering the chance and impact of erroneous judgements.

Computer Science and Game Theory,Computers and Society,General Economics
Promoting an open research culture

B. A. Nosek,G. Alter,G. C. Banks,D. Borsboom,S. D. Bowman,S. J. Breckler,S. Buck,C. D. Chambers,G. Chin,G. Christensen,M. Contestabile,A. Dafoe,E. Eich,J. Freese,R. Glennerster,D. Goroff,D. P. Green,B. Hesse,M. Humphreys,J. Ishiyama,D. Karlan,A. Kraut,A. Lupia,P. Mabry,T. Madon,N. Malhotra,E. Mayo-Wilson,M. McNutt,E. Miguel,E. Levy Paluck,U. Simonsohn,C. Soderberg,B. A. Spellman,J. Turitto,G. VandenBos,S. Vazire,E. J. Wagenmakers,R. Wilson,T. Yarkoni

DOI: https://doi.org/10.1126/science.aab2374

IF: 56.9

2015-06-26

Science

Abstract:Author guidelines for journals could help to promote transparency, openness, and reproducibility

multidisciplinary sciences
A harm reduction approach to improving peer review by acknowledging its imperfections

Steven J. Cooke,Nathan Young,Kathryn S. Peiman,Dominique G. Roche,Jeff C. Clements,Andrew N. Kadykalo,Jennifer F. Provencher,Rajeev Raghavan,Maria C. DeRosa,Robert J. Lennox,Aminah Robinson Fayek,Melania E. Cristescu,Stuart J. Murray,Joanna Quinn,Kelly D. Cobey,Howard I. Browman

DOI: https://doi.org/10.1139/facets-2024-0102

2024-09-19

FACETS

Abstract:FACETS, Volume 9, Issue , Page 1-14, January 2024. This candid perspective written by scholars from diverse disciplinary backgrounds is intended to advance conversations about the realities of peer review and its inherent limitations. Trust in a process or institution is built slowly and can be destroyed quickly. Trust in the peer review process for scholarly outputs (i.e., journal articles) is being eroded by high-profile scandals, exaggerated news stories, exposés, corrections, retractions, and anecdotes about poor practices. Diminished trust in the peer review process has real-world consequences and threatens the uptake of critical scientific advances. The literature on "crises of trust" tells us that rebuilding diminished trust takes time and requires frank admission and discussion of problems, creative thinking that addresses rather than dismisses criticisms, and planning and enacting short- and long-term reforms to address the root causes of problems. This article takes steps in this direction by presenting eight peer review reality checks and summarizing efforts to address their weaknesses using a harm reduction approach, though we recognize that reforms take time and some problems may never be fully rectified. While some forms of harm reduction will require structural and procedural changes, we emphasize the vital role that training editors, reviewers, and authors has in harm reduction. Additionally, consumers of science need training about how the peer review process works and how to critically evaluate research findings. No amount of self-policing, transparency, or reform to peer review will eliminate all bad actors, unscrupulous publishers, perverse incentives that reward cutting corners, intentional deception, or bias. However, the scientific community can act to minimize the harms from these activities, while simultaneously (re)building the peer review process. A peer review system is needed, even if it is imperfect.

multidisciplinary sciences
Avoiding a Tragedy of the Commons in the Peer Review Process

D Sculley,Jasper Snoek,Alex Wiltschko

DOI: https://doi.org/10.48550/arXiv.1901.06246

2018-12-18

Abstract:Peer review is the foundation of scientific publication, and the task of reviewing has long been seen as a cornerstone of professional service. However, the massive growth in the field of machine learning has put this community benefit under stress, threatening both the sustainability of an effective review process and the overall progress of the field. In this position paper, we argue that a tragedy of the commons outcome may be avoided by emphasizing the professional aspects of this service. In particular, we propose a rubric to hold reviewers to an objective standard for review quality. In turn, we also propose that reviewers be given appropriate incentive. As one possible such incentive, we explore the idea of financial compensation on a per-review basis. We suggest reasonable funding models and thoughts on long term effects.

Computers and Society,Digital Libraries,Machine Learning
Recommendations for accelerating open preprint peer review to improve the culture of science

Michele Avissar-Whiting,Frédérique Belliard,Stefano M. Bertozzi,Amy Brand,Katherine Brown,Géraldine Clément-Stoneham,Stephanie Dawson,Gautam Dey,Daniel Ecer,Scott C. Edmunds,Ashley Farley,Tara D. Fischer,Maryrose Franko,James S. Fraser,Kathryn Funk,Clarisse Ganier,Melissa Harrison,Anna Hatch,Haley Hazlett,Samantha Hindle,Daniel W. Hook,Phil Hurst,Sophien Kamoun,Robert Kiley,Michael M. Lacy,Marcel LaFlamme,Rebecca Lawrence,Thomas Lemberger,Maria Leptin,Elliott Lumb,Catriona J. MacCallum,Christopher Steven Marcum,Gabriele Marinello,Alex Mendonça,Sara Monaco,Kleber Neves,Damian Pattinson,Jessica K. Polka,Iratxe Puebla,Martyn Rittman,Stephen J. Royle,Daniela Saderi,Richard Sever,Kathleen Shearer,John E. Spiro,Bodo Stern,Dario Taraborelli,Ron Vale,Claudia G. Vasquez,Ludo Waltman,Fiona M. Watt,Zara Y. Weinberg,Mark Williams

DOI: https://doi.org/10.1371/journal.pbio.3002502

IF: 9.8

2024-03-02

PLoS Biology

Abstract:Peer review is an important part of the scientific process, but traditional peer review at journals is coming under increased scrutiny for its inefficiency and lack of transparency. As preprints become more widely used and accepted, they raise the possibility of rethinking the peer-review process. Preprints are enabling new forms of peer review that have the potential to be more thorough, inclusive, and collegial than traditional journal peer review, and to thus fundamentally shift the culture of peer review toward constructive collaboration. In this Consensus View, we make a call to action to stakeholders in the community to accelerate the growing momentum of preprint sharing and provide recommendations to empower researchers to provide open and constructive peer review for preprints.

biochemistry & molecular biology,biology
Views on the peer review system of biomedical journals: an online survey of academics from high-ranking universities

Roger Chun-Man Ho,Kwok-Kei Mak,Ren Tao,Yanxia Lu,Jeffrey R Day,Fang Pan

DOI: https://doi.org/10.1186/1471-2288-13-74

2013-06-07

BMC Medical Research Methodology

Abstract:Abstract Background Peer review is the major method used by biomedical journals for making the decision of publishing an article. This cross-sectional survey assesses views concerning the review system of biomedical journals among academics globally. Methods A total of 28,009 biomedical academics from high-ranking universities listed by the 2009 Times Higher Education Quacquarelli Symonds (THE-QS) World University Rankings were contacted by email between March 2010 and August 2010. 1,340 completed an online survey which focused on their academic background, negative experiences and views on biomedical journal peer review and the results were compared among basic scientists, clinicians and clinician scientists. Results Fewer than half of the respondents agreed that the peer review systems of biomedical journals were fair (48.4%), scientific (47.5%), or transparent (25.1%). Nevertheless, 58.2% of the respondents agreed that authors should remain anonymous and 64.4% agreed that reviewers should not be disclosed. Most, (67.7%) agreed to the establishment of an appeal system. The proportion of native English-speaking respondents who agreed that the “peer review system is fair” was significantly higher than for non-native respondents (p = 0.02). Similarly, the proportion of clinicians stating that the “peer review system is fair” was significantly higher than that for basic scientists and clinician-scientists (p = 0.004). For females, (β = −0.1, p = 0.03), the frequency of encountering personal attacks in reviewers’ comments (β = −0.1, p = 0.002) and the frequency of imposition of unnecessary references by reviewers (β = −0.06, p = 0.04) were independently and inversely associated with agreement that “the peer review system is fair”. Conclusion Academics are divided on the issue of whether the biomedical journal peer review system is fair, scientific and transparent. A majority of academics agreed with the double-blind peer review and to the establishment of an appeal system. Female academics, experience of personal attacks and imposition of unnecessary references by reviewers were related to disagreement about fairness of the peer review system of biomedical journals.

health care sciences & services
The Peer Review Process: Past, Present, and Future

John A. Drozdz,Michael R. Ladomery

DOI: https://doi.org/10.3389/bjbs.2024.12054

2024-06-17

British Journal of Biomedical Science

Abstract:The peer review process is a fundamental aspect of modern scientific paper publishing, underpinning essential quality control. First conceptualised in the 1700s, it is an iterative process that aims to elevate scientific literature to the highest standards whilst preventing publication of scientifically unsound, potentially misleading, and even plagiarised information. It is widely accepted that the peer review of scientific papers is an irreplaceable and fundamental aspect of the research process. However, the rapid growth of research and technology has led to a huge increase in the number of publications. This has led to increased pressure on the peer review system. There are several established peer review methodologies, ranging from single and double blind to open and transparent review, but their implementation across journals and research fields varies greatly. Some journals are testing entirely novel approaches (such as collaborative reviews), whilst others are piloting changes to established methods. Given the unprecedented growth in publication numbers, and the ensuing burden on journals, editors, and reviewers, it is imperative to improve the quality and efficiency of the peer review process. Herein we evaluate the peer review process, from its historical origins to current practice and future directions.

medical laboratory technology
A multi-disciplinary perspective on emergent and future innovations in peer review

Jonathan P Tennant,Jonathan M Dugan,Daniel Graziotin,Damien C Jacques,François Waldner,Daniel Mietchen,Yehia Elkhatib,Lauren B Collister,Christina K Pikas,Tom Crick,Paola Masuzzo,Anthony Caravaggi,Devin R Berg,Kyle E Niemeyer,Tony Ross-Hellauer,Sara Mannheimer,Lillian Rigling,Daniel S Katz,Bastian Greshake Tzovaras,Josmel Pacheco-Mendoza,Nazeefa Fatima,Marta Poblet,Marios Isaakidis,Dasapta Erwin Irawan,Sébastien Renaut,Christopher R Madan,Lisa Matthias,Jesper Nørgaard Kjær,Daniel Paul O'Donnell,Cameron Neylon,Sarah Kearns,Manojkumar Selvaraju,Julien Colomb

DOI: https://doi.org/10.12688/f1000research.12037.3

2017-07-20

Abstract:Peer review of research articles is a core part of our scholarly communication system. In spite of its importance, the status and purpose of peer review is often contested. What is its role in our modern digital research and communications infrastructure? Does it perform to the high standards with which it is generally regarded? Studies of peer review have shown that it is prone to bias and abuse in numerous dimensions, frequently unreliable, and can fail to detect even fraudulent research. With the advent of web technologies, we are now witnessing a phase of innovation and experimentation in our approaches to peer review. These developments prompted us to examine emerging models of peer review from a range of disciplines and venues, and to ask how they might address some of the issues with our current systems of peer review. We examine the functionality of a range of social Web platforms, and compare these with the traits underlying a viable peer review system: quality control, quantified performance metrics as engagement incentives, and certification and reputation. Ideally, any new systems will demonstrate that they out-perform and reduce the biases of existing models as much as possible. We conclude that there is considerable scope for new peer review initiatives to be developed, each with their own potential issues and advantages. We also propose a novel hybrid platform model that could, at least partially, resolve many of the socio-technical issues associated with peer review, and potentially disrupt the entire scholarly communication system. Success for any such development relies on reaching a critical threshold of research community engagement with both the process and the platform, and therefore cannot be achieved without a significant change of incentives in research environments.
Safeguarding Scientific Integrity: Examining Conflicts of Interest in the Peer Review Process

Leslie D. McIntosh,Cynthia Hudson Vitale

2023-08-08

Abstract:This case study analyzes the expertise, potential conflicts of interest, and objectivity of editors, authors, and peer reviewers involved in a 2022 special journal issue on fertility, pregnancy, and mental health. Data were collected on qualifications, organizational affiliations, and relationships among six papers' authors, three guest editors, and twelve peer reviewers. Two articles were found to have undisclosed conflicts of interest between authors, an editor, and multiple peer reviewers affiliated with anti-abortion advocacy and lobbying groups, indicating compromised objectivity. This lack of transparency undermines the peer review process and enables biased research and disinformation proliferation. To increase integrity, we recommend multiple solutions: open peer review, expanded conflict of interest disclosure, increased stakeholder accountability, and retraction when ethical standards are violated. By illuminating noncompliance with ethical peer review guidelines, this study aims to raise awareness to help prevent the propagation of partisan science through respected scholarly channels.

Digital Libraries
The How and Why of Peer Review

Sarah H. Kagan

DOI: https://doi.org/10.1111/opn.12625

2024-06-25

International Journal of Older People Nursing

Abstract:International Journal of Older People Nursing, Volume 19, Issue 4, July 2024.

nursing,geriatrics & gerontology,gerontology
In praise of the referee

Nicolas Chopin,Andrew Gelman,Kerrie L. Mengersen,Christian P. Robert

DOI: https://doi.org/10.48550/arXiv.1205.4304

2012-05-19

Abstract:There has been a lively debate in many fields, including statistics and related applied fields such as psychology and biomedical research, on possible reforms of the scholarly publishing system. Currently, referees contribute so much to improve scientific papers, both directly through constructive criticism and indirectly through the threat of rejection. We discuss ways in which new approaches to journal publication could continue to make use of the valuable efforts of peer reviewers.

Other Statistics,Digital Libraries,Physics and Society
A Randomized Controlled Trial on Anonymizing Reviewers to Each Other in Peer Review Discussions

Charvi Rastogi,Xiangchen Song,Zhijing Jin,Ivan Stelmakh,Hal Daumé III,Kun Zhang,Nihar B. Shah

2024-03-02

Abstract:Peer review often involves reviewers submitting their independent reviews, followed by a discussion among reviewers of each paper. A question among policymakers is whether the reviewers of a paper should be anonymous to each other during the discussion. We shed light on this by conducting a randomized controlled trial at the UAI 2022 conference. We randomly split the reviewers and papers into two conditions--one with anonymous discussions and the other with non-anonymous discussions, and conduct an anonymous survey of all reviewers, to address the following questions: 1. Do reviewers discuss more in one of the conditions? Marginally more in anonymous (n = 2281, p = 0.051). 2. Does seniority have more influence on final decisions when non-anonymous? Yes, the decisions are closer to senior reviewers' scores in the non-anonymous condition than in anonymous (n = 484, p = 0.04). 3. Are reviewers more polite in one of the conditions? No significant difference in politeness of reviewers' text-based responses (n = 1125, p = 0.72). 4. Do reviewers' self-reported experiences differ across the two conditions? No significant difference for each of the five questions asked (n = 132 and p > 0.3). 5. Do reviewers prefer one condition over the other? Yes, there is a weak preference for anonymous discussions (n = 159 and Cohen's d= 0.25). 6. What do reviewers consider important to make policy on anonymity among reviewers? Reviewers' feeling of safety in expressing their opinions was rated most important, while polite communication among reviewers was rated least important (n = 159). 7. Have reviewers experienced dishonest behavior due to non-anonymity in discussions? Yes, roughly 7% of respondents answered affirmatively (n = 167). Overall, this experiment reveals evidence supporting an anonymous discussion setup in the peer-review process, in terms of the evaluation criteria considered.

Computers and Society,Digital Libraries
Peer Review and Medical Journals

Kenneth Nugent,Christopher J Peterson

DOI: https://doi.org/10.1177/21501319241252235

Abstract:Journal editors depend on peer reviewers to make decisions about submitted manuscripts. These reviewers help evaluate the methods, the results, the discussion of the results, and the overall organization and presentation of the manuscript. In addition, reviewers can help identify important mistakes and possible misconduct. Editors frequently have difficulty obtaining enough peer reviews which are submitted in a timely manner. This increases the workload of editors and journal managers and potentially delays the publication of clinical and research studies. This commentary discusses of the importance of peer reviews and make suggestions which potentially can increase the participation of academic faculty and researchers in this important activity.
Beyond Journals and Peer Review: Towards a More Flexible Ecosystem for Scholarly Communication

Michael Wood

DOI: https://doi.org/10.20944/preprints202012.0612.v2

2021-08-30

Abstract:This article challenges the assumption that journals and peer review are essential for developing, evaluating and disseminating scientific and other academic knowledge. It suggests a more flexible ecosystem, and examines some of the possibilities this might facilitate. The market for academic outputs should be opened up by encouraging the separation of the dissemination service from the evaluation service. Publishing research in subject-specific journals encourages compartmentalising research into rigid categories. The dissemination of knowledge would be better served by an open access, web-based repository system encompassing all disciplines. There would then be a role for organisations to assess the items in this repository to help users find relevant, high-quality work. There could be a variety of such organisations which could enable reviews from peers to be supplemented with evaluation by non-peers from a variety of different perspectives: user reviews, statistical reviews, reviews from the perspective of different disciplines, and so on. This should reduce the inevitably conservative influence of relying on two or three peers, and make the evaluation system more critical, multi-dimensional and responsive to the requirements of different audience groups, changing circumstances, and new ideas. Non-peer review might make it easier to challenge dominant paradigms, and expanding the potential audience beyond a narrow group of peers might encourage the criterion of simplicity to be taken more seriously - which is essential if human knowledge is to continue to progress.
Peer review demystified: part 1

DOI: https://doi.org/10.1038/s41592-024-02225-6

IF: 48

2024-03-13

Nature Methods

Abstract:Peer review is at the heart of publishing scientific papers. In this first installment of a two-part Editorial, we explain how we manage the process at Nature Methods.

biochemical research methods
Acknowledgment to Reviewers of Brain Sciences in 2021

DOI: https://doi.org/10.3390/brainsci12020171

IF: 3.333

2022-01-28

Brain Sciences

Abstract:Rigorous peer-reviews are the basis of high-quality academic publishing [...]

neurosciences
Peer review by committee? New journal rethinks old model

Jackson Ryan

DOI: https://doi.org/10.1038/d41586-024-03039-4

IF: 64.8

2024-10-02

Nature

Abstract:Teams of up to seven researchers are reviewing papers submitted to the Stacks Journal, which wants to bring evaluation of scientific work back from the brink. Teams of up to seven researchers are reviewing papers submitted to the Stacks Journal, which wants to bring evaluation of scientific work back from the brink.

multidisciplinary sciences
What Constitutes Peer Review of Data: A survey of published peer review guidelines

Todd A Carpenter

DOI: https://doi.org/10.48550/arXiv.1704.02236

2017-04-07

Abstract:Since a number of journals specifically focus on the review and publication of data sets, reviewing their policies seems an appropriate place to start in assessing what existing practice looks like in the 'real world' of reviewing and publishing data. This article outlines a study of the publicly available peer review policies of 39 scientific publications that publish data papers to discern which criteria are most and least frequently referenced. It also compares current practice with proposed criteria published in 2012.

Digital Libraries
Thank you to reviewers

Peter Allebeck,Diana Delnoij,Róza Ádány,Stefania Boccia,Alastair Leyland

DOI: https://doi.org/10.1093/eurpub/ckae007

2024-02-29

European Journal of Public Health

Abstract:A scientific journal exists only because of the good will and assistance of its reviewers. We can only publish approximately 25% of the papers we receive. In order to publish really high-quality papers, we depend on the expertise of many people. In 2023, a total of 406 reviewers provided their expert opinions for us to decide about the papers that we should publish. Their names are listed below, and we would like to take this opportunity to thank them.

public, environmental & occupational health

Publish peer reviews

Motivating and Maintaining Ethics, Equity, Effectiveness, Efficiency, and Expertise in Peer Review

Decentralized Peer Review in Open Science: A Mechanism Proposal

Promoting an open research culture

A harm reduction approach to improving peer review by acknowledging its imperfections

Avoiding a Tragedy of the Commons in the Peer Review Process

Recommendations for accelerating open preprint peer review to improve the culture of science

Views on the peer review system of biomedical journals: an online survey of academics from high-ranking universities

The Peer Review Process: Past, Present, and Future

A multi-disciplinary perspective on emergent and future innovations in peer review

Safeguarding Scientific Integrity: Examining Conflicts of Interest in the Peer Review Process

The How and Why of Peer Review

In praise of the referee

A Randomized Controlled Trial on Anonymizing Reviewers to Each Other in Peer Review Discussions

Peer Review and Medical Journals

Beyond Journals and Peer Review: Towards a More Flexible Ecosystem for Scholarly Communication

Peer review demystified: part 1

Acknowledgment to Reviewers of Brain Sciences in 2021

Peer review by committee? New journal rethinks old model

What Constitutes Peer Review of Data: A survey of published peer review guidelines

Thank you to reviewers