Revisiting the Why of Peer Review: Sustainability of the Current System

Sameh Eltaybani

DOI: https://doi.org/10.1111/opn.12642

2024-09-03

International Journal of Older People Nursing

Abstract:International Journal of Older People Nursing, Volume 19, Issue 5, September 2024.

nursing,geriatrics & gerontology,gerontology

What problem does this paper attempt to address?

The How and Why of Peer Review

Sarah H. Kagan

DOI: https://doi.org/10.1111/opn.12625

2024-06-25

International Journal of Older People Nursing

Abstract:International Journal of Older People Nursing, Volume 19, Issue 4, July 2024.

nursing,geriatrics & gerontology,gerontology
Peer review in academic publishing: Challenges in achieving the gold standard

Prue Gonzalez,Gail Wilson,Alison Purvis

DOI: https://doi.org/10.53761/1.19.5.1

2022-07-02

Journal of University Teaching and Learning Practice

Abstract:This editorial describes some of the current and emerging challenges in peer review for the academic publishing system. Peer review is a fundamental element of academic research and publishing, with a firm reliance on the global scholarly community to perform gatekeeping and filtering processes in pursuing high-quality and high-value scholarly publications: the “gold standard” in academic publishing. We begin with examples of several contemporary challenges the peer review system poses, including impartiality and bias, academic reward structures, fake peer reviews, and reviewer fatigue. To further understand these challenges, we then provide a brief history of the evolution of the peer review system, focusing on the traditional forms of pre-publication peer review so familiar to the communication of scholarly work. Against this backdrop, we consider the benefits of peer review that span the continuum of the academic community – from authors to reviewers to journals and research communities. But many traditional forms of peer review are being challenged by new and innovative processes, systems and platforms. Finally, we look at how others have re-envisioned the peer review process during this phase of rapid evolution in journal publishing, with a strong call for quality and integrity in writing peer reviews. We conclude by suggesting ways forward for embedding sustainability, equity, and respect within the peer review process as an active force for advancing scholarship.
A multi-disciplinary perspective on emergent and future innovations in peer review

Jonathan P Tennant,Jonathan M Dugan,Daniel Graziotin,Damien C Jacques,François Waldner,Daniel Mietchen,Yehia Elkhatib,Lauren B Collister,Christina K Pikas,Tom Crick,Paola Masuzzo,Anthony Caravaggi,Devin R Berg,Kyle E Niemeyer,Tony Ross-Hellauer,Sara Mannheimer,Lillian Rigling,Daniel S Katz,Bastian Greshake Tzovaras,Josmel Pacheco-Mendoza,Nazeefa Fatima,Marta Poblet,Marios Isaakidis,Dasapta Erwin Irawan,Sébastien Renaut,Christopher R Madan,Lisa Matthias,Jesper Nørgaard Kjær,Daniel Paul O'Donnell,Cameron Neylon,Sarah Kearns,Manojkumar Selvaraju,Julien Colomb

DOI: https://doi.org/10.12688/f1000research.12037.3

2017-07-20

Abstract:Peer review of research articles is a core part of our scholarly communication system. In spite of its importance, the status and purpose of peer review is often contested. What is its role in our modern digital research and communications infrastructure? Does it perform to the high standards with which it is generally regarded? Studies of peer review have shown that it is prone to bias and abuse in numerous dimensions, frequently unreliable, and can fail to detect even fraudulent research. With the advent of web technologies, we are now witnessing a phase of innovation and experimentation in our approaches to peer review. These developments prompted us to examine emerging models of peer review from a range of disciplines and venues, and to ask how they might address some of the issues with our current systems of peer review. We examine the functionality of a range of social Web platforms, and compare these with the traits underlying a viable peer review system: quality control, quantified performance metrics as engagement incentives, and certification and reputation. Ideally, any new systems will demonstrate that they out-perform and reduce the biases of existing models as much as possible. We conclude that there is considerable scope for new peer review initiatives to be developed, each with their own potential issues and advantages. We also propose a novel hybrid platform model that could, at least partially, resolve many of the socio-technical issues associated with peer review, and potentially disrupt the entire scholarly communication system. Success for any such development relies on reaching a critical threshold of research community engagement with both the process and the platform, and therefore cannot be achieved without a significant change of incentives in research environments.
Motivating and Maintaining Ethics, Equity, Effectiveness, Efficiency, and Expertise in Peer Review

Adam Craig,Christina Lee,Nithyaa Bala,Carl Taswell,

DOI: https://doi.org/10.48085/i5b147d9d

2022-06-30

Brainiacs Journal of Brain Imaging And Computing Sciences

Abstract:Scientists who engage in science and the scientific endeavor should seek truth with conviction of morals and commitment to ethics. While the number of publications continues to increase, the number of retractions has increased at a faster rate. Journals publish fraudulent research papers despite claims of peer review and adherence to publishing ethics. Nevertheless, appropriate ethical peer review will remain a gatekeeper when selecting research manuscripts in scholarly publishing and approving research applications for grant funding. However, this peer review must become more open, fair, transparent, equitable, and just with new recommendations and guidelines for reproducible and accountable reviews that support and promote fair citation and citational justice. We should engineer this new peer-review process with modern informatics technology and information science to provide and defend better safeguards for truth and integrity, to clarify and maintain the provenance of information and ideas, and to rebuild and restore trust in scholarly research institutions. Indeed, this new approach will be necessary in the current post-truth era to counter the ease and speed with which mis-information, dis-information, anti-information, caco-information, and mal-information spread through the internet, web, news, and social media. The most important question for application of new peer-review methods to these information wars should be ‘Who does what when?’ in support of reproducible and accountable reviews. Who refers to the authors, reviewers, editors, and publishers as participants in the review process. What refers to disclosure of the participants' identities, the material content of author manuscripts and reviewer commentaries, and other communications between authors and reviewers. When refers to tracking the sequential points in time for which disclosure of whose identity, which content, and which communication at which step of the peer-review process for which audience of readers and reviewers. We believe that quality peer review, and peer review of peer review, must be motivated and maintained by elevating their status and prestige to an art and a science. Both peer review itself and peer review analyses of peer reviews should be incentivised by publishing peer reviews as citable references separately from the research report reviewed while crossreferenced and crosslinked to the report reviewed.
The critical role of peer reviewers: Challenges and future steps

Nady El-Guebaly,John Foster,Anees Bahji,Matilda Hellman

DOI: https://doi.org/10.1177/14550725221092862

Abstract:The critical role of peer reviewers in the publishing process is examined. Examples of typical challenges are provided, including the relative lack of rewards for this important task. Particular attention is paid to the need to consider the diversity of the peer reviewers recruited and impediments to the selection beyond Areas of Competence, often due to the small available pool. Finally, recommendations for improvement are suggested.
The Peer Review Process: Past, Present, and Future

John A. Drozdz,Michael R. Ladomery

DOI: https://doi.org/10.3389/bjbs.2024.12054

2024-06-17

British Journal of Biomedical Science

Abstract:The peer review process is a fundamental aspect of modern scientific paper publishing, underpinning essential quality control. First conceptualised in the 1700s, it is an iterative process that aims to elevate scientific literature to the highest standards whilst preventing publication of scientifically unsound, potentially misleading, and even plagiarised information. It is widely accepted that the peer review of scientific papers is an irreplaceable and fundamental aspect of the research process. However, the rapid growth of research and technology has led to a huge increase in the number of publications. This has led to increased pressure on the peer review system. There are several established peer review methodologies, ranging from single and double blind to open and transparent review, but their implementation across journals and research fields varies greatly. Some journals are testing entirely novel approaches (such as collaborative reviews), whilst others are piloting changes to established methods. Given the unprecedented growth in publication numbers, and the ensuing burden on journals, editors, and reviewers, it is imperative to improve the quality and efficiency of the peer review process. Herein we evaluate the peer review process, from its historical origins to current practice and future directions.

medical laboratory technology
Evaluating alternative systems of peer review: a large-scale agent-based modelling approach to scientific publication

Michail Kovanis,Ludovic Trinquart,Philippe Ravaud,Raphaël Porcher

DOI: https://doi.org/10.1007/s11192-017-2375-1

IF: 3.801

2017-04-03

Scientometrics

Abstract:The debate on whether the peer-review system is in crisis has been heated recently. A variety of alternative systems have been proposed to improve the system and make it sustainable. However, we lack sufficient evidence and data related to these issues. Here we used a previously developed agent-based model of the scientific publication and peer-review system calibrated with empirical data to compare the efficiency of five alternative peer-review systems with the conventional system. We modelled two systems of immediate publication, with and without online reviews (crowdsourcing), a system with only one round of reviews and revisions allowed (re-review opt-out) and two review-sharing systems in which rejected manuscripts are resubmitted along with their past reviews to any other journal (portable) or to only those of the same publisher but of lower impact factor (cascade). The review-sharing systems outperformed or matched the performance of the conventional one in all peer-review efficiency, reviewer effort and scientific dissemination metrics we used. The systems especially showed a large decrease in total time of the peer-review process and total time devoted by reviewers to complete all reports in a year. The two systems with immediate publication released more scientific information than the conventional one but provided almost no other benefit. Re-review opt-out decreased the time reviewers devoted to peer review but had lower performance on screening papers that should not be published and relative increase in intrinsic quality of papers due to peer review than the conventional system. Sensitivity analyses showed consistent findings to those from our main simulations. We recommend prioritizing a system of review-sharing to create a sustainable scientific publication and peer-review system.

information science & library science,computer science, interdisciplinary applications
The limitations to our understanding of peer review

Jonathan P. Tennant,Tony Ross-Hellauer

DOI: https://doi.org/10.1186/s41073-020-00092-1

2020-04-30

Research Integrity and Peer Review

Abstract:Abstract Peer review is embedded in the core of our knowledge generation systems, perceived as a method for establishing quality or scholarly legitimacy for research, while also often distributing academic prestige and standing on individuals. Despite its critical importance, it curiously remains poorly understood in a number of dimensions. In order to address this, we have analysed peer review to assess where the major gaps in our theoretical and empirical understanding of it lie. We identify core themes including editorial responsibility, the subjectivity and bias of reviewers, the function and quality of peer review, and the social and epistemic implications of peer review. The high-priority gaps are focused around increased accountability and justification in decision-making processes for editors and developing a deeper, empirical understanding of the social impact of peer review. Addressing this at the bare minimum will require the design of a consensus for a minimal set of standards for what constitutes peer review, and the development of a shared data infrastructure to support this. Such a field requires sustained funding and commitment from publishers and research funders, who both have a commitment to uphold the integrity of the published scholarly record. We use this to present a guide for the future of peer review, and the development of a new research discipline based on the study of peer review.
Avoiding a Tragedy of the Commons in the Peer Review Process

D Sculley,Jasper Snoek,Alex Wiltschko

DOI: https://doi.org/10.48550/arXiv.1901.06246

2018-12-18

Abstract:Peer review is the foundation of scientific publication, and the task of reviewing has long been seen as a cornerstone of professional service. However, the massive growth in the field of machine learning has put this community benefit under stress, threatening both the sustainability of an effective review process and the overall progress of the field. In this position paper, we argue that a tragedy of the commons outcome may be avoided by emphasizing the professional aspects of this service. In particular, we propose a rubric to hold reviewers to an objective standard for review quality. In turn, we also propose that reviewers be given appropriate incentive. As one possible such incentive, we explore the idea of financial compensation on a per-review basis. We suggest reasonable funding models and thoughts on long term effects.

Computers and Society,Digital Libraries,Machine Learning
A harm reduction approach to improving peer review by acknowledging its imperfections

Steven J. Cooke,Nathan Young,Kathryn S. Peiman,Dominique G. Roche,Jeff C. Clements,Andrew N. Kadykalo,Jennifer F. Provencher,Rajeev Raghavan,Maria C. DeRosa,Robert J. Lennox,Aminah Robinson Fayek,Melania E. Cristescu,Stuart J. Murray,Joanna Quinn,Kelly D. Cobey,Howard I. Browman

DOI: https://doi.org/10.1139/facets-2024-0102

2024-09-19

FACETS

Abstract:FACETS, Volume 9, Issue , Page 1-14, January 2024. This candid perspective written by scholars from diverse disciplinary backgrounds is intended to advance conversations about the realities of peer review and its inherent limitations. Trust in a process or institution is built slowly and can be destroyed quickly. Trust in the peer review process for scholarly outputs (i.e., journal articles) is being eroded by high-profile scandals, exaggerated news stories, exposés, corrections, retractions, and anecdotes about poor practices. Diminished trust in the peer review process has real-world consequences and threatens the uptake of critical scientific advances. The literature on "crises of trust" tells us that rebuilding diminished trust takes time and requires frank admission and discussion of problems, creative thinking that addresses rather than dismisses criticisms, and planning and enacting short- and long-term reforms to address the root causes of problems. This article takes steps in this direction by presenting eight peer review reality checks and summarizing efforts to address their weaknesses using a harm reduction approach, though we recognize that reforms take time and some problems may never be fully rectified. While some forms of harm reduction will require structural and procedural changes, we emphasize the vital role that training editors, reviewers, and authors has in harm reduction. Additionally, consumers of science need training about how the peer review process works and how to critically evaluate research findings. No amount of self-policing, transparency, or reform to peer review will eliminate all bad actors, unscrupulous publishers, perverse incentives that reward cutting corners, intentional deception, or bias. However, the scientific community can act to minimize the harms from these activities, while simultaneously (re)building the peer review process. A peer review system is needed, even if it is imperfect.

multidisciplinary sciences
Journal peer review: a bar or bridge? An analysis of a paper’s revision history and turnaround time, and the effect on citation

J. Rigby,D. Cox,K. Julian

DOI: https://doi.org/10.1007/s11192-017-2630-5

IF: 3.801

2018-01-03

Scientometrics

Abstract:Journal peer review lies at the heart of academic quality control. This article explores the journal peer review process and seeks to examine how the reviewing process might itself contribute to papers, leading them to be more highly cited and to achieve greater recognition. Our work builds on previous observations and views expressed in the literature about (a) the role of actors involved in the research and publication process that suggest that peer review is inherent in the research process and (b) on the contribution reviewers themselves might make to the content and increased citation of papers. Using data from the journal peer review process of a single journal in the Social Sciences field (Business, Management and Accounting), we examine the effects of peer review on papers submitted to that journal including the effect upon citation, a novel step in the study of the outcome of peer review. Our detailed analysis suggests, contrary to initial assumptions, that it is not the time taken to revise papers but the actual number of revisions that leads to greater recognition for papers in terms of citation impact. Our study provides evidence, albeit limited to the case of a single journal, that the peer review process may constitute a form of knowledge production and is not the simple correction of errors contained in submitted papers.

information science & library science,computer science, interdisciplinary applications
Peer Review as an Evolving Response to Organizational Constraint: Evidence from Sociology Journals, 1952–2018

Ben Merriman

DOI: https://doi.org/10.1007/s12108-020-09473-x

2020-11-18

The American Sociologist

Abstract:Double-blind peer review is a central feature of the editorial model of most journals in sociology and neighboring social scientific fields, yet there is little history of how and when its main features developed. Drawing from nearly 70 years of annual reports of the editors of American Sociological Association journals, this article describes the historical emergence of major elements of editorial peer review. These reports and associated descriptive statistics are used to show that blind review, ad hoc review, the formal requirement of exclusive submission, routine use of the revise and resubmit decision, and common use of desk rejection developed separately over a period of decades. The article then argues that the ongoing evolution of the review model has not been driven by intellectual considerations. Rather, the evolution of peer review is best understood as the product of continuous efforts to steward editors' scarce attention while preserving an open submission policy that favors authors' interests.
How to improve scientific peer review: Four schools of thought

Ludo Waltman,Wolfgang Kaltenbrunner,Stephen Pinfield,Helen Buckley Woods

DOI: https://doi.org/10.1002/leap.1544

Abstract:Peer review plays an essential role as one of the cornerstones of the scholarly publishing system. There are many initiatives that aim to improve the way in which peer review is organized, resulting in a highly complex landscape of innovation in peer review. Different initiatives are based on different views on the most urgent challenges faced by the peer review system, leading to a diversity of perspectives on how the system can be improved. To provide a more systematic understanding of the landscape of innovation in peer review, we suggest that the landscape is shaped by four schools of thought: The Quality & Reproducibility school, the Democracy & Transparency school, the Equity & Inclusion school, and the Efficiency & Incentives school. Each school has a different view on the key problems of the peer review system and the innovations necessary to address these problems. The schools partly complement each other, but we argue that there are also important tensions between them. We hope that the four schools of thought offer a useful framework to facilitate conversations about the future development of the peer review system.
Time to Rethink Academic Publishing: the Peer Reviewer Crisis.

Carolina Tropini,B Brett Finlay,Mark Nichter,Melissa K Melby,Jessica L Metcalf,Maria Gloria Dominguez-Bello,Liping Zhao,Margaret J McFall-Ngai,Naama Geva-Zatorsky,Katherine R Amato,Eduardo A Undurraga,Hendrik N Poinar,Jack A Gilbert

DOI: https://doi.org/10.1128/mbio.01091-23

IF: 6.4

2023-01-01

mBio

Abstract:There is concern that the time taken to publish academic papers in microbiological science has significantly increased in recent years. While the data do not specifically support this, evidence suggests that editors are having to invite more and more reviewers to identify those willing to perform peer review.
Acknowledgment to Reviewers of Applied System Innovation in 2020

Applied System Innovation Editorial Office

DOI: https://doi.org/10.3390/asi4010007

2021-01-26

Applied System Innovation

Abstract:Peer review is the driving force of journal development, and reviewers are gatekeepers who ensure that Applied System Innovation maintains its standards for the high quality of its published papers [...]
Peer Reviewing Revisited: Assessing Research with Interlinked Semantic Comments

Cristina-Iulia Bucur,Tobias Kuhn,Davide Ceolin

DOI: https://doi.org/10.48550/arXiv.1910.03218

2019-10-08

Abstract:Scientific publishing seems to be at a turning point. Its paradigm has stayed basically the same for 300 years but is now challenged by the increasing volume of articles that makes it very hard for scientists to stay up to date in their respective fields. In fact, many have pointed out serious flaws of current scientific publishing practices, including the lack of accuracy and efficiency of the reviewing process. To address some of these problems, we apply here the general principles of the Web and the Semantic Web to scientific publishing, focusing on the reviewing process. We want to determine if a fine-grained model of the scientific publishing workflow can help us make the reviewing processes better organized and more accurate, by ensuring that review comments are created with formal links and semantics from the start. Our contributions include a novel model called Linkflows that allows for such detailed and semantically rich representations of reviews and the reviewing processes. We evaluate our approach on a manually curated dataset from several recent Computer Science journals and conferences that come with open peer reviews. We gathered ground-truth data by contacting the original reviewers and asking them to categorize their own review comments according to our model. Comparing this ground truth to answers provided by model experts, peers, and automated techniques confirms that our approach of formally capturing the reviewers' intentions from the start prevents substantial discrepancies compared to when this information is later extracted from the plain-text comments. In general, our analysis shows that our model is well understood and easy to apply, and it revealed the semantic properties of such review comments.

Digital Libraries
Where have all the reviewers gone?

Rachel H. Ellaway

DOI: https://doi.org/10.1007/s10459-024-10350-2

IF: 3.629

2024-06-14

Advances in Health Sciences Education

Abstract:In this editorial the editor considers the growing challenges journals are facing in securing peer reviewers, some of the approaches being tried to address this problem, and the prospects for sustaining communities of scholars with and without an ongoing commitment to peer review.

education & educational research,education, scientific disciplines,health care sciences & services
On The Peer Review Reports: Does Size Matter?

Abdelghani Maddi,Luis Miotti

DOI: https://doi.org/10.1007/s11192-024-04977-6

2024-03-07

Abstract:Amidst the ever-expanding realm of scientific production and the proliferation of predatory journals, the focus on peer review remains paramount for scientometricians and sociologists of science. Despite this attention, there is a notable scarcity of empirical investigations into the tangible impact of peer review on publication quality. This study aims to address this gap by conducting a comprehensive analysis of how peer review contributes to the quality of scholarly publications, as measured by the citations they receive.Utilizing an adjusted dataset comprising 57,482 publications from Publons to Web of Science and employing the Raking Ratio method, our study reveals intriguing insights. Specifically, our findings shed light on a nuanced relationship between the length of reviewer reports and the subsequent citations received by publications. Through a robust regression analysis, we establish that, beginning from 947 words, the length of reviewer reports is significantly associated with an increase in citations.These results not only confirm the initial hypothesis that longer reports indicate requested improvements, thereby enhancing the quality and visibility of articles, but also underscore the importance of timely and comprehensive reviewer reports. Furthermore, insights from Publons' data suggest that open access to reports can influence reviewer behavior, encouraging more detailed reports.Beyond the scholarly landscape, our findings prompt a reevaluation of the role of reviewers, emphasizing the need to recognize and value this resource-intensive yet underappreciated activity in institutional evaluations. Additionally, the study sounds a cautionary note regarding the challenges faced by peer review in the context of an increasing volume of submissions, potentially compromising the vigilance of peers in swiftly assessing numerous articles.

Digital Libraries
Ancient texts to PubMed: a brief history of the peer-review process

P R Farrell,L Magida Farrell,M K Farrell

DOI: https://doi.org/10.1038/jp.2016.209

2016-11-17

Journal of Perinatology

Abstract:The formal evaluation of scientific literature by invited referees (peer reviewers) is a relatively recent phenomenon and now is considered a cornerstone of modern science. However, its roots can be traced back to antiquity. As the speed and complexity of scientific information and publishing increases in the digital age, peer review must continue to evolve. To understand the future direction of peer review, we must understand its past. Here, we briefly explore the history of scientific peer review. This may help us predict and design appropriate peer review for the new era. This work was originally presented at the Pediatric Academic Societies Annual Meeting in Baltimore, Maryland in the Spring of 2016.

pediatrics,obstetrics & gynecology
A history and development of peer-review process

Siladitya Jana

2019-07-12

Abstract:The paper shows the importance of peer review process in the scholarly communication system and discusses both the closed and the newly emerging open peer review models. It also examines the peer review system at the scholarly academies or societies in their nomination systems for prizes, rewards, etc. It also discusses the various facets of the newly developed open peer review models now prevalent in various journals. The paper may help to understand and appreciate the role played by peer review in the scholarly communication system and the efforts being made to make it more transparent.

History,Education,Political Science

Revisiting the Why of Peer Review: Sustainability of the Current System

The How and Why of Peer Review

Peer review in academic publishing: Challenges in achieving the gold standard

A multi-disciplinary perspective on emergent and future innovations in peer review

Motivating and Maintaining Ethics, Equity, Effectiveness, Efficiency, and Expertise in Peer Review

The critical role of peer reviewers: Challenges and future steps

The Peer Review Process: Past, Present, and Future

Evaluating alternative systems of peer review: a large-scale agent-based modelling approach to scientific publication

The limitations to our understanding of peer review

Avoiding a Tragedy of the Commons in the Peer Review Process

A harm reduction approach to improving peer review by acknowledging its imperfections

Journal peer review: a bar or bridge? An analysis of a paper’s revision history and turnaround time, and the effect on citation

Peer Review as an Evolving Response to Organizational Constraint: Evidence from Sociology Journals, 1952–2018

How to improve scientific peer review: Four schools of thought

Time to Rethink Academic Publishing: the Peer Reviewer Crisis.

Acknowledgment to Reviewers of Applied System Innovation in 2020

Peer Reviewing Revisited: Assessing Research with Interlinked Semantic Comments

Where have all the reviewers gone?

On The Peer Review Reports: Does Size Matter?

Ancient texts to PubMed: a brief history of the peer-review process

A history and development of peer-review process