Rejection, Rebuttal, Revision - Some Flexible Features of Peer-Review

DB RUBIN

DOI: https://doi.org/10.1017/s0140525x00011638

1982-01-01

Behavioral and Brain Sciences

Abstract:An abstract is not available for this content so a preview has been provided. Please use the Get access link above for information on how to access this content.

What problem does this paper attempt to address?

Does My Rebuttal Matter? Insights from a Major NLP Conference

Yang Gao,Steffen Eger,Ilia Kuznetsov,Iryna Gurevych,Yusuke Miyao

DOI: https://doi.org/10.48550/arXiv.1903.11367

2019-03-28

Abstract:Peer review is a core element of the scientific process, particularly in conference-centered fields such as ML and NLP. However, only few studies have evaluated its properties empirically. Aiming to fill this gap, we present a corpus that contains over 4k reviews and 1.2k author responses from ACL-2018. We quantitatively and qualitatively assess the corpus. This includes a pilot study on paper weaknesses given by reviewers and on quality of author responses. We then focus on the role of the rebuttal phase, and propose a novel task to predict after-rebuttal (i.e., final) scores from initial reviews and author responses. Although author responses do have a marginal (and statistically significant) influence on the final scores, especially for borderline papers, our results suggest that a reviewer's final score is largely determined by her initial score and the distance to the other reviewers' initial scores. In this context, we discuss the conformity bias inherent to peer reviewing, a bias that has largely been overlooked in previous research. We hope our analyses will help better assess the usefulness of the rebuttal phase in NLP conferences.

Computation and Language,Artificial Intelligence
A harm reduction approach to improving peer review by acknowledging its imperfections

Steven J. Cooke,Nathan Young,Kathryn S. Peiman,Dominique G. Roche,Jeff C. Clements,Andrew N. Kadykalo,Jennifer F. Provencher,Rajeev Raghavan,Maria C. DeRosa,Robert J. Lennox,Aminah Robinson Fayek,Melania E. Cristescu,Stuart J. Murray,Joanna Quinn,Kelly D. Cobey,Howard I. Browman

DOI: https://doi.org/10.1139/facets-2024-0102

2024-09-19

FACETS

Abstract:FACETS, Volume 9, Issue , Page 1-14, January 2024. This candid perspective written by scholars from diverse disciplinary backgrounds is intended to advance conversations about the realities of peer review and its inherent limitations. Trust in a process or institution is built slowly and can be destroyed quickly. Trust in the peer review process for scholarly outputs (i.e., journal articles) is being eroded by high-profile scandals, exaggerated news stories, exposés, corrections, retractions, and anecdotes about poor practices. Diminished trust in the peer review process has real-world consequences and threatens the uptake of critical scientific advances. The literature on "crises of trust" tells us that rebuilding diminished trust takes time and requires frank admission and discussion of problems, creative thinking that addresses rather than dismisses criticisms, and planning and enacting short- and long-term reforms to address the root causes of problems. This article takes steps in this direction by presenting eight peer review reality checks and summarizing efforts to address their weaknesses using a harm reduction approach, though we recognize that reforms take time and some problems may never be fully rectified. While some forms of harm reduction will require structural and procedural changes, we emphasize the vital role that training editors, reviewers, and authors has in harm reduction. Additionally, consumers of science need training about how the peer review process works and how to critically evaluate research findings. No amount of self-policing, transparency, or reform to peer review will eliminate all bad actors, unscrupulous publishers, perverse incentives that reward cutting corners, intentional deception, or bias. However, the scientific community can act to minimize the harms from these activities, while simultaneously (re)building the peer review process. A peer review system is needed, even if it is imperfect.

multidisciplinary sciences
Peer review and journal models

Paolo Dall'Aglio

DOI: https://doi.org/10.48550/arXiv.physics/0608307

2006-08-31

Physics and Society

Abstract:Tentative analysis of alternative peer review and journal models. Open Access and Impact factor issues are not covered. The bibliography, in alphabetical order, lists articles and web sites (with brief description).
The Peer Review Process: Past, Present, and Future

John A. Drozdz,Michael R. Ladomery

DOI: https://doi.org/10.3389/bjbs.2024.12054

2024-06-17

British Journal of Biomedical Science

Abstract:The peer review process is a fundamental aspect of modern scientific paper publishing, underpinning essential quality control. First conceptualised in the 1700s, it is an iterative process that aims to elevate scientific literature to the highest standards whilst preventing publication of scientifically unsound, potentially misleading, and even plagiarised information. It is widely accepted that the peer review of scientific papers is an irreplaceable and fundamental aspect of the research process. However, the rapid growth of research and technology has led to a huge increase in the number of publications. This has led to increased pressure on the peer review system. There are several established peer review methodologies, ranging from single and double blind to open and transparent review, but their implementation across journals and research fields varies greatly. Some journals are testing entirely novel approaches (such as collaborative reviews), whilst others are piloting changes to established methods. Given the unprecedented growth in publication numbers, and the ensuing burden on journals, editors, and reviewers, it is imperative to improve the quality and efficiency of the peer review process. Herein we evaluate the peer review process, from its historical origins to current practice and future directions.

medical laboratory technology
True Peer Review

Amit K. Chopra

DOI: https://doi.org/10.48550/arXiv.1209.2807

2012-09-13

Abstract:In computer science, conferences and journals conduct peer review in order to decide what to publish. Many have pointed out the inherent weaknesses in peer review, including those of bias, quality, and accountability. Many have suggested and adopted refinements of peer review, for instance, double blind peer review with author rebuttals. In this essay, I argue that peer review as currently practiced conflates the sensible idea of getting comments on a paper with the irrevocably-flawed one that we either accept or reject the paper, which I term gatekeeping. If we look at the two separately, then it is clear that the ills associated with current peer review systems are not due to the practice of getting comments, but due to the practice of gatekeeping. True peer review constitutes my proposal for replacing existing peer review systems. It embraces the idea of open debate on the merits of a paper; however, it rejects unequivocally the exercise of gatekeeping. True peer review offers all the benefits of current peer review systems but has none of its weaknesses. True peer review will lead to a truly engaged community of researchers and therefore better science.

Digital Libraries
Learnings From 1000 Rejections

Alex Edmans

DOI: https://doi.org/10.1111/fima.12487

2024-11-16

Financial Management

Abstract:The Review of Finance aimed to significantly increase its standards over my 6 years as managing editor and 1 year as editor. To comply with these new standards, I had to reject nearly 1000 manuscripts. This paper aims to use these rejections constructively by distilling common reasons for rejection to guide future research. They are divided into three categories: contribution, execution, and exposition. Beyond extracts from decision letters that give reasons for rejection, this paper also shares excerpts that shed light on the editorial process, such as how an editor weighs up feedback to reach a decision, as well as emails to authors outside formal letters in response to queries on the process.

business, finance
Revise and Resubmit: An Intertextual Model of Text-based Collaboration in Peer Review

Ilia Kuznetsov,Jan Buchmann,Max Eichler,Iryna Gurevych

DOI: https://doi.org/10.48550/arXiv.2204.10805

2022-04-22

Computation and Language

Abstract:Peer review is a key component of the publishing process in most fields of science. The increasing submission rates put a strain on reviewing quality and efficiency, motivating the development of applications to support the reviewing and editorial work. While existing NLP studies focus on the analysis of individual texts, editorial assistance often requires modeling interactions between pairs of texts -- yet general frameworks and datasets to support this scenario are missing. Relationships between texts are the core object of the intertextuality theory -- a family of approaches in literary studies not yet operationalized in NLP. Inspired by prior theoretical work, we propose the first intertextual model of text-based collaboration, which encompasses three major phenomena that make up a full iteration of the review-revise-and-resubmit cycle: pragmatic tagging, linking and long-document version alignment. While peer review is used across the fields of science and publication formats, existing datasets solely focus on conference-style review in computer science. Addressing this, we instantiate our proposed model in the first annotated multi-domain corpus in journal-style post-publication open peer review, and provide detailed insights into the practical aspects of intertextual annotation. Our resource is a major step towards multi-domain, fine-grained applications of NLP in editorial support for peer review, and our intertextual framework paves the path for general-purpose modeling of text-based collaboration. Our corpus and accompanying code are publicly available.
QRB Volume 11 Issue 3 Back Matter

Robert Berger,Carole R. Sawicki,Quentin H. Gibson,Eric D. Crandall,Abdullah Y. Obaid,Roy S. Small,Perturbations -A Schechter Chairman,Richard W. Fessenden,Nishant Verma,King T,Ruben Maas,Markus Gassner,R. K. Nanda,Willis Conover,Oleg Jardetzky,Persoons,L Hellemans,Margit M. K. Nass,Helen Lester,Mary Beth Krouse,S Vuk-Pavlovi,Yoav Blatt,Doron Lancet,Israel Pecht,Cornelis P.J. Glaudemans,Robin R. Austin,Sander Chan,Marc Antonini,B Brunori,Paola Giardina,Giuseppe Benedetti,Stefano Bianchini,Marion Grassi,Radu Alexandru Badea,Ludwig Detoma,Jeffrey D. Brand,F Lakowicz,Perturbations -L Large,Mitchell R. Dorfman,Ivan Simić,Taub,A Schulten,Weller,Marc Faraggi,L Klapper,N Dorfman,R Alberding,Stephen F. Austin,L. Chan,H Eisenstein,D Frauenfelder,Kathryn L. Good,Michael Kaufmann,T Marden,L Nordlund,August Reinisch,Laurie Reynolds,G. Sørensen,Karen Dineen Wagner,Yansheng Liang,S Rentzepis,D. G. H. Ballard,N Mauzerall,C Geacintov,A Swenberg,Rodrigo Ramírez Campillo,S.W. Hyer,Kenneth Shapiro,Michael Andreas Winn

DOI: https://doi.org/10.1017/s0033583500002274

1978-01-01

Quarterly Reviews of Biophysics

Abstract:An abstract is not available for this content so a preview has been provided. As you have access to this content, a full PDF is available via the ‘Save PDF’ action button.
A multi-disciplinary perspective on emergent and future innovations in peer review

Jonathan P Tennant,Jonathan M Dugan,Daniel Graziotin,Damien C Jacques,François Waldner,Daniel Mietchen,Yehia Elkhatib,Lauren B Collister,Christina K Pikas,Tom Crick,Paola Masuzzo,Anthony Caravaggi,Devin R Berg,Kyle E Niemeyer,Tony Ross-Hellauer,Sara Mannheimer,Lillian Rigling,Daniel S Katz,Bastian Greshake Tzovaras,Josmel Pacheco-Mendoza,Nazeefa Fatima,Marta Poblet,Marios Isaakidis,Dasapta Erwin Irawan,Sébastien Renaut,Christopher R Madan,Lisa Matthias,Jesper Nørgaard Kjær,Daniel Paul O'Donnell,Cameron Neylon,Sarah Kearns,Manojkumar Selvaraju,Julien Colomb

DOI: https://doi.org/10.12688/f1000research.12037.3

2017-07-20

Abstract:Peer review of research articles is a core part of our scholarly communication system. In spite of its importance, the status and purpose of peer review is often contested. What is its role in our modern digital research and communications infrastructure? Does it perform to the high standards with which it is generally regarded? Studies of peer review have shown that it is prone to bias and abuse in numerous dimensions, frequently unreliable, and can fail to detect even fraudulent research. With the advent of web technologies, we are now witnessing a phase of innovation and experimentation in our approaches to peer review. These developments prompted us to examine emerging models of peer review from a range of disciplines and venues, and to ask how they might address some of the issues with our current systems of peer review. We examine the functionality of a range of social Web platforms, and compare these with the traits underlying a viable peer review system: quality control, quantified performance metrics as engagement incentives, and certification and reputation. Ideally, any new systems will demonstrate that they out-perform and reduce the biases of existing models as much as possible. We conclude that there is considerable scope for new peer review initiatives to be developed, each with their own potential issues and advantages. We also propose a novel hybrid platform model that could, at least partially, resolve many of the socio-technical issues associated with peer review, and potentially disrupt the entire scholarly communication system. Success for any such development relies on reaching a critical threshold of research community engagement with both the process and the platform, and therefore cannot be achieved without a significant change of incentives in research environments.
A Proposal for the Establishment of Review Boards - a flexible approach to the selection of academic knowledge

Bruce Edmonds

DOI: https://doi.org/10.48550/arXiv.cs/9904001

1999-04-01

Computers and Society

Abstract:Paper journals use a small number of trusted academics to select information on behalf of all their readers. This inflexibility in the selection was justified due to the expense of publishing. The advent of cheap distribution via the internet allows a new trade-off between time and expense and the flexibility of the selection process. This paper explores one such possible process one where the role of mark-up and archiving is separated from that of review. The idea is that authors publish their papers on their own web pages or in a public paper archive, a board of reviewers judge that paper on a number of different criteria. The detailed results of the reviews are stored in such a way as to enable readers to use these judgements to find the papers they want using search engines on the web. Thus instead of journals using generic selection criteria readers can set their own to suit their needs. The resulting system might be even cheaper than web-journals to implement.
Reputation, review, . . . and other stories

DOI: https://doi.org/10.1136/bmj.o2568

2022-11-04

The BMJ

Abstract:Biases of peer reviewAuthors' reputations have a strong influence on peer reviewers' recommendations, according to the results of an experiment in which 3000 researchers were invited to comment on a paper jointly written by a Nobel laureate and a junior research associate. When the junior researcher was the only name shown on the manuscript, 65% of reviewers recommended rejection. When the more famous moniker was revealed, only 23% of reviewers wanted to reject the paper (SSRN doi:10.2139/ssrn.4190976).Salt content of foodSurveys of packaged food products sold in the UK show that, between 2015 and 2020, mean salt content fell by a nugatory 4%. The reduction wasn't statistically significant and might represent no more than random variation over time (PLOS Med doi:10.1371/journal.pmed.1004114). It looks as if voluntary salt reduction targets for the UK food industry are having little effect, which is disappointing since reducing salt intake is a declared public health priority.Stages...
In praise of the referee

Nicolas Chopin,Andrew Gelman,Kerrie L. Mengersen,Christian P. Robert

DOI: https://doi.org/10.48550/arXiv.1205.4304

2012-05-19

Abstract:There has been a lively debate in many fields, including statistics and related applied fields such as psychology and biomedical research, on possible reforms of the scholarly publishing system. Currently, referees contribute so much to improve scientific papers, both directly through constructive criticism and indirectly through the threat of rejection. We discuss ways in which new approaches to journal publication could continue to make use of the valuable efforts of peer reviewers.

Other Statistics,Digital Libraries,Physics and Society
The limitations to our understanding of peer review

Jonathan P. Tennant,Tony Ross-Hellauer

DOI: https://doi.org/10.1186/s41073-020-00092-1

2020-04-30

Research Integrity and Peer Review

Abstract:Abstract Peer review is embedded in the core of our knowledge generation systems, perceived as a method for establishing quality or scholarly legitimacy for research, while also often distributing academic prestige and standing on individuals. Despite its critical importance, it curiously remains poorly understood in a number of dimensions. In order to address this, we have analysed peer review to assess where the major gaps in our theoretical and empirical understanding of it lie. We identify core themes including editorial responsibility, the subjectivity and bias of reviewers, the function and quality of peer review, and the social and epistemic implications of peer review. The high-priority gaps are focused around increased accountability and justification in decision-making processes for editors and developing a deeper, empirical understanding of the social impact of peer review. Addressing this at the bare minimum will require the design of a consensus for a minimal set of standards for what constitutes peer review, and the development of a shared data infrastructure to support this. Such a field requires sustained funding and commitment from publishers and research funders, who both have a commitment to uphold the integrity of the published scholarly record. We use this to present a guide for the future of peer review, and the development of a new research discipline based on the study of peer review.
Peer review demystified: part 2

DOI: https://doi.org/10.1038/s41592-024-02258-x

IF: 48

2024-04-13

Nature Methods

Abstract:Nature Methods, Published online: 12 April 2024; doi:10.1038/s41592-024-02258-x We continue our explanation of the peer review process at Nature Methods.

biochemical research methods
Beyond Journals and Peer Review: Towards a More Flexible Ecosystem for Scholarly Communication

Michael Wood

DOI: https://doi.org/10.20944/preprints202012.0612.v2

2021-08-30

Abstract:This article challenges the assumption that journals and peer review are essential for developing, evaluating and disseminating scientific and other academic knowledge. It suggests a more flexible ecosystem, and examines some of the possibilities this might facilitate. The market for academic outputs should be opened up by encouraging the separation of the dissemination service from the evaluation service. Publishing research in subject-specific journals encourages compartmentalising research into rigid categories. The dissemination of knowledge would be better served by an open access, web-based repository system encompassing all disciplines. There would then be a role for organisations to assess the items in this repository to help users find relevant, high-quality work. There could be a variety of such organisations which could enable reviews from peers to be supplemented with evaluation by non-peers from a variety of different perspectives: user reviews, statistical reviews, reviews from the perspective of different disciplines, and so on. This should reduce the inevitably conservative influence of relying on two or three peers, and make the evaluation system more critical, multi-dimensional and responsive to the requirements of different audience groups, changing circumstances, and new ideas. Non-peer review might make it easier to challenge dominant paradigms, and expanding the potential audience beyond a narrow group of peers might encourage the criterion of simplicity to be taken more seriously - which is essential if human knowledge is to continue to progress.
A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications

Dongyeop Kang,Waleed Ammar,Bhavana Dalvi,Madeleine van Zuylen,Sebastian Kohlmeier,Eduard Hovy,Roy Schwartz

DOI: https://doi.org/10.48550/arXiv.1804.09635

2018-04-25

Computation and Language

Abstract:Peer reviewing is a central component in the scientific publishing process. We present the first public dataset of scientific peer reviews available for research purposes (PeerRead v1) providing an opportunity to study this important artifact. The dataset consists of 14.7K paper drafts and the corresponding accept/reject decisions in top-tier venues including ACL, NIPS and ICLR. The dataset also includes 10.7K textual peer reviews written by experts for a subset of the papers. We describe the data collection process and report interesting observed phenomena in the peer reviews. We also propose two novel NLP tasks based on this dataset and provide simple baseline models. In the first task, we show that simple models can predict whether a paper is accepted with up to 21% error reduction compared to the majority baseline. In the second task, we predict the numerical scores of review aspects and show that simple models can outperform the mean baseline for aspects with high variance such as 'originality' and 'impact'.
TrueReview: A Platform for Post-Publication Peer Review

Luca de Alfaro,Marco Faella

DOI: https://doi.org/10.48550/arXiv.1608.07878

2016-08-29

Digital Libraries

Abstract:In post-publication peer review, scientific contributions are first published in open-access forums, such as arXiv or other digital libraries, and are subsequently reviewed and possibly ranked and/or evaluated. Compared to the classical process of scientific publishing, in which review precedes publication, post-publication peer review leads to faster dissemination of ideas, and publicly-available reviews. The chief concern in post-publication reviewing consists in eliciting high-quality, insightful reviews from participants. We describe the mathematical foundations and structure of TrueReview, an open-source tool we propose to build in support of post-publication review. In TrueReview, the motivation to review is provided via an incentive system that promotes reviews and evaluations that are both truthful (they turn out to be correct in the long run) and informative (they provide significant new information). TrueReview organizes papers in venues, allowing different scientific communities to set their own submission and review policies. These venues can be manually set-up, or they can correspond to categories in well-known repositories such as arXiv. The review incentives can be used to form a reviewer ranking that can be prominently displayed alongside papers in the various disciplines, thus offering a concrete benefit to reviewers. The paper evaluations, in turn, reward the authors of the most significant papers, both via an explicit paper ranking, and via increased visibility in search.
Decoding the Rejection Code: Understanding Why Articles Get Axed

Vishal V Bhende,Tanishq S Sharma,Mathangi Krishnakumar,Anikode Subramanian Ramaswamy,Kanchan Bilgi,Saptak P Mankad

DOI: https://doi.org/10.7759/cureus.56920

2024-03-25

Cureus

Abstract:In the competitive arena of medical publishing, manuscript rejection remains a significant barrier to disseminating research findings. This editorial delves into the multifaceted nature of manuscript rejection, elucidating common reasons and proposing actionable strategies for authors to enhance their chances of acceptance. Key rejection factors include a mismatch with journal scope, lack of novelty, methodological flaws, inconclusive results, ethical issues, poor presentation, data inaccessibility, author misconduct, and plagiarism. Ethical lapses, such as lacking informed consent, or submissions fraught with grammatical errors, further doom manuscripts. In addressing these pitfalls, authors are advised to ensure content originality, methodological rigor, ethical compliance, and clear presentation. Aligning the manuscript with the journal's audience, scope, and editorial standards is crucial, as is professional conduct and responsiveness to feedback. Leveraging technological tools for citation management, grammar checking, and plagiarism detection can also significantly bolster manuscript quality. Ultimately, understanding and addressing common rejection reasons can empower authors to improve their submissions, contributing to the advancement of medical knowledge and their professional growth.
Is Peer Review a Good Idea?

Remco Heesen,Liam Kofi Bright

DOI: https://doi.org/10.1093/bjps/axz029

2021-09-01

The British Journal for the Philosophy of Science

Abstract:Prepublication peer review should be abolished. We consider the effects that such a change will have on the social structure of science, paying particular attention to the changed incentive structure and the likely effects on the behaviour of individual scientists. We evaluate these changes from the perspective of epistemic consequentialism. We find that where the effects of abolishing prepublication peer review can be evaluated with a reasonable level of confidence based on presently available evidence, they are either positive or neutral. We conclude that on present evidence abolishing peer review weakly dominates the status quo.1. Introduction2. Setting the Stage3. Benefits of Abolishing Peer Review3.1. Sharing scientific results3.2. Time allocation3.3. Gender skew in publications3.4. Library resources3.5. Scientific careers3.6. The power of gatekeepers4. Where Peer Review Makes No Difference4.1. Epistemic sorting4.2. Malpractice detection4.3. Herding behaviour4.4. Long-run credit5. Difficulties for Our Proposal5.1. A guarantee for outsiders5.2. A runaway Matthew effect6. Conclusion
Journal peer review: a bar or bridge? An analysis of a paper’s revision history and turnaround time, and the effect on citation

J. Rigby,D. Cox,K. Julian

DOI: https://doi.org/10.1007/s11192-017-2630-5

IF: 3.801

2018-01-03

Scientometrics

Abstract:Journal peer review lies at the heart of academic quality control. This article explores the journal peer review process and seeks to examine how the reviewing process might itself contribute to papers, leading them to be more highly cited and to achieve greater recognition. Our work builds on previous observations and views expressed in the literature about (a) the role of actors involved in the research and publication process that suggest that peer review is inherent in the research process and (b) on the contribution reviewers themselves might make to the content and increased citation of papers. Using data from the journal peer review process of a single journal in the Social Sciences field (Business, Management and Accounting), we examine the effects of peer review on papers submitted to that journal including the effect upon citation, a novel step in the study of the outcome of peer review. Our detailed analysis suggests, contrary to initial assumptions, that it is not the time taken to revise papers but the actual number of revisions that leads to greater recognition for papers in terms of citation impact. Our study provides evidence, albeit limited to the case of a single journal, that the peer review process may constitute a form of knowledge production and is not the simple correction of errors contained in submitted papers.

information science & library science,computer science, interdisciplinary applications

Rejection, Rebuttal, Revision - Some Flexible Features of Peer-Review

Does My Rebuttal Matter? Insights from a Major NLP Conference

A harm reduction approach to improving peer review by acknowledging its imperfections

Peer review and journal models

The Peer Review Process: Past, Present, and Future

True Peer Review

Learnings From 1000 Rejections

Revise and Resubmit: An Intertextual Model of Text-based Collaboration in Peer Review

QRB Volume 11 Issue 3 Back Matter

A multi-disciplinary perspective on emergent and future innovations in peer review

A Proposal for the Establishment of Review Boards - a flexible approach to the selection of academic knowledge

Reputation, review, . . . and other stories

In praise of the referee

The limitations to our understanding of peer review

Peer review demystified: part 2

Beyond Journals and Peer Review: Towards a More Flexible Ecosystem for Scholarly Communication

A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications

TrueReview: A Platform for Post-Publication Peer Review

Decoding the Rejection Code: Understanding Why Articles Get Axed

Is Peer Review a Good Idea?

Journal peer review: a bar or bridge? An analysis of a paper’s revision history and turnaround time, and the effect on citation