Peer review and journal models

Paolo Dall'Aglio

DOI: https://doi.org/10.48550/arXiv.physics/0608307

2006-08-31

Physics and Society

Abstract:Tentative analysis of alternative peer review and journal models. Open Access and Impact factor issues are not covered. The bibliography, in alphabetical order, lists articles and web sites (with brief description).

What problem does this paper attempt to address?

Evaluating alternative systems of peer review: a large-scale agent-based modelling approach to scientific publication

Michail Kovanis,Ludovic Trinquart,Philippe Ravaud,Raphaël Porcher

DOI: https://doi.org/10.1007/s11192-017-2375-1

IF: 3.801

2017-04-03

Scientometrics

Abstract:The debate on whether the peer-review system is in crisis has been heated recently. A variety of alternative systems have been proposed to improve the system and make it sustainable. However, we lack sufficient evidence and data related to these issues. Here we used a previously developed agent-based model of the scientific publication and peer-review system calibrated with empirical data to compare the efficiency of five alternative peer-review systems with the conventional system. We modelled two systems of immediate publication, with and without online reviews (crowdsourcing), a system with only one round of reviews and revisions allowed (re-review opt-out) and two review-sharing systems in which rejected manuscripts are resubmitted along with their past reviews to any other journal (portable) or to only those of the same publisher but of lower impact factor (cascade). The review-sharing systems outperformed or matched the performance of the conventional one in all peer-review efficiency, reviewer effort and scientific dissemination metrics we used. The systems especially showed a large decrease in total time of the peer-review process and total time devoted by reviewers to complete all reports in a year. The two systems with immediate publication released more scientific information than the conventional one but provided almost no other benefit. Re-review opt-out decreased the time reviewers devoted to peer review but had lower performance on screening papers that should not be published and relative increase in intrinsic quality of papers due to peer review than the conventional system. Sensitivity analyses showed consistent findings to those from our main simulations. We recommend prioritizing a system of review-sharing to create a sustainable scientific publication and peer-review system.

information science & library science,computer science, interdisciplinary applications
Complex systems approach to scientific publication and peer-review system: development of an agent-based model calibrated with empirical journal data

Michail Kovanis,Raphaël Porcher,Philippe Ravaud,Ludovic Trinquart

DOI: https://doi.org/10.1007/s11192-015-1800-6

IF: 3.801

2015-12-10

Scientometrics

Abstract:Scientific peer-review and publication systems incur a huge burden in terms of costs and time. Innovative alternatives have been proposed to improve the systems, but assessing their impact in experimental studies is not feasible at a systemic level. We developed an agent-based model by adopting a unified view of peer review and publication systems and calibrating it with empirical journal data in the biomedical and life sciences. We modeled researchers, research manuscripts and scientific journals as agents. Researchers were characterized by their scientific level and resources, manuscripts by their scientific value, and journals by their reputation and acceptance or rejection thresholds. These state variables were used in submodels for various processes such as production of articles, submissions to target journals, in-house and external peer review, and resubmissions. We collected data for a sample of biomedical and life sciences journals regarding acceptance rates, resubmission patterns and total number of published articles. We adjusted submodel parameters so that the agent-based model outputs fit these empirical data. We simulated 105 journals, 25,000 researchers and 410,000 manuscripts over 10 years. A mean of 33,600 articles were published per year; 19 % of submitted manuscripts remained unpublished. The mean acceptance rate was 21 % after external peer review and rejection rate 32 % after in-house review; 15 % publications resulted from the first submission, 47 % the second submission and 20 % the third submission. All decisions in the model were mainly driven by the scientific value, whereas journal targeting and persistence in resubmission defined whether a manuscript would be published or abandoned after one or many rejections. This agent-based model may help in better understanding the determinants of the scientific publication and peer-review systems. It may also help in assessing and identifying the most promising alternative systems of peer review.

information science & library science,computer science, interdisciplinary applications
A multi-disciplinary perspective on emergent and future innovations in peer review

Jonathan P Tennant,Jonathan M Dugan,Daniel Graziotin,Damien C Jacques,François Waldner,Daniel Mietchen,Yehia Elkhatib,Lauren B Collister,Christina K Pikas,Tom Crick,Paola Masuzzo,Anthony Caravaggi,Devin R Berg,Kyle E Niemeyer,Tony Ross-Hellauer,Sara Mannheimer,Lillian Rigling,Daniel S Katz,Bastian Greshake Tzovaras,Josmel Pacheco-Mendoza,Nazeefa Fatima,Marta Poblet,Marios Isaakidis,Dasapta Erwin Irawan,Sébastien Renaut,Christopher R Madan,Lisa Matthias,Jesper Nørgaard Kjær,Daniel Paul O'Donnell,Cameron Neylon,Sarah Kearns,Manojkumar Selvaraju,Julien Colomb

DOI: https://doi.org/10.12688/f1000research.12037.3

2017-07-20

Abstract:Peer review of research articles is a core part of our scholarly communication system. In spite of its importance, the status and purpose of peer review is often contested. What is its role in our modern digital research and communications infrastructure? Does it perform to the high standards with which it is generally regarded? Studies of peer review have shown that it is prone to bias and abuse in numerous dimensions, frequently unreliable, and can fail to detect even fraudulent research. With the advent of web technologies, we are now witnessing a phase of innovation and experimentation in our approaches to peer review. These developments prompted us to examine emerging models of peer review from a range of disciplines and venues, and to ask how they might address some of the issues with our current systems of peer review. We examine the functionality of a range of social Web platforms, and compare these with the traits underlying a viable peer review system: quality control, quantified performance metrics as engagement incentives, and certification and reputation. Ideally, any new systems will demonstrate that they out-perform and reduce the biases of existing models as much as possible. We conclude that there is considerable scope for new peer review initiatives to be developed, each with their own potential issues and advantages. We also propose a novel hybrid platform model that could, at least partially, resolve many of the socio-technical issues associated with peer review, and potentially disrupt the entire scholarly communication system. Success for any such development relies on reaching a critical threshold of research community engagement with both the process and the platform, and therefore cannot be achieved without a significant change of incentives in research environments.
Accelerating the pace of discovery by changing the peer review algorithm

Stefano Allesina

DOI: https://doi.org/10.48550/arXiv.0911.0344

2009-11-03

Abstract:The number of scientific publications is constantly rising, increasing the strain on the review process. The number of submissions is actually higher, as each manuscript is often reviewed several times before publication. To face the deluge of submissions, top journals reject a considerable fraction of manuscripts without review, potentially declining manuscripts with merit. The situation is frustrating for authors, reviewers and editors alike. Recently, several editors wrote about the ``tragedy of the reviewer commons', advocating for urgent corrections to the system. Almost every scientist has ideas on how to improve the system, but it is very difficult, if not impossible, to perform experiments to test which measures would be most effective. Surprisingly, relatively few attempts have been made to model peer review. Here I implement a simulation framework in which ideas on peer review can be quantitatively tested. I incorporate authors, reviewers, manuscripts and journals into an agent-based model and a peer review system emerges from their interactions. As a proof-of-concept, I contrast an implementation of the current system, in which authors decide the journal for their submissions, with a system in which journals bid on manuscripts for publication. I show that, all other things being equal, this latter system solves most of the problems currently associated with the peer review process. Manuscripts' evaluation is faster, authors publish more and in better journals, and reviewers' effort is optimally utilized. However, more work is required from editors. This modeling framework can be used to test other solutions for peer review, leading the way for an improvement of how science is disseminated.

Digital Libraries,Physics and Society,Other Quantitative Biology
Peer-review in the Internet age

Pawel Sobkowicz

DOI: https://doi.org/10.48550/arXiv.0810.0486

2008-10-03

Abstract:The importance of peer-review in the scientific process can not be overestimated. Yet, due to increasing pressures of research and exponentially growing number of publications the task faced by the referees becomes ever more difficult. We discuss here a few possible improvements that would enable more efficient review of the scientific literature, using the growing Internet connectivity. In particular, a practical automated model for providing the referees with references to papers that might have strong relationship with the work under review, based on general network properties of citations is proposed.

Physics and Society,Digital Libraries
Peer review vs bibliometrics: which method better predicts the scholarly impact of publications?

Giovanni Abramo,Ciriaco Andrea D'Angelo,Emanuela Reale

DOI: https://doi.org/10.1007/s11192-019-03184-y

2019-09-13

Abstract:In this work, we try to answer the question of which method, peer review vs bibliometrics, better predicts the future overall scholarly impact of scientific publications. We measure the agreement between peer review evaluations of Web of Science indexed publications submitted to the first Italian research assessment exercise and long-term citations of the same publications. We do the same for an early citation-based indicator. We find that the latter shows stronger predictive power, i.e., it more reliably predicts late citations in all the disciplinary areas examined, and for any citation time window starting one year after publication.

Digital Libraries
The Peer Review Process: Past, Present, and Future

John A. Drozdz,Michael R. Ladomery

DOI: https://doi.org/10.3389/bjbs.2024.12054

2024-06-17

British Journal of Biomedical Science

Abstract:The peer review process is a fundamental aspect of modern scientific paper publishing, underpinning essential quality control. First conceptualised in the 1700s, it is an iterative process that aims to elevate scientific literature to the highest standards whilst preventing publication of scientifically unsound, potentially misleading, and even plagiarised information. It is widely accepted that the peer review of scientific papers is an irreplaceable and fundamental aspect of the research process. However, the rapid growth of research and technology has led to a huge increase in the number of publications. This has led to increased pressure on the peer review system. There are several established peer review methodologies, ranging from single and double blind to open and transparent review, but their implementation across journals and research fields varies greatly. Some journals are testing entirely novel approaches (such as collaborative reviews), whilst others are piloting changes to established methods. Given the unprecedented growth in publication numbers, and the ensuing burden on journals, editors, and reviewers, it is imperative to improve the quality and efficiency of the peer review process. Herein we evaluate the peer review process, from its historical origins to current practice and future directions.

medical laboratory technology
Peer Reviewing Revisited: Assessing Research with Interlinked Semantic Comments

Cristina-Iulia Bucur,Tobias Kuhn,Davide Ceolin

DOI: https://doi.org/10.48550/arXiv.1910.03218

2019-10-08

Abstract:Scientific publishing seems to be at a turning point. Its paradigm has stayed basically the same for 300 years but is now challenged by the increasing volume of articles that makes it very hard for scientists to stay up to date in their respective fields. In fact, many have pointed out serious flaws of current scientific publishing practices, including the lack of accuracy and efficiency of the reviewing process. To address some of these problems, we apply here the general principles of the Web and the Semantic Web to scientific publishing, focusing on the reviewing process. We want to determine if a fine-grained model of the scientific publishing workflow can help us make the reviewing processes better organized and more accurate, by ensuring that review comments are created with formal links and semantics from the start. Our contributions include a novel model called Linkflows that allows for such detailed and semantically rich representations of reviews and the reviewing processes. We evaluate our approach on a manually curated dataset from several recent Computer Science journals and conferences that come with open peer reviews. We gathered ground-truth data by contacting the original reviewers and asking them to categorize their own review comments according to our model. Comparing this ground truth to answers provided by model experts, peers, and automated techniques confirms that our approach of formally capturing the reviewers' intentions from the start prevents substantial discrepancies compared to when this information is later extracted from the plain-text comments. In general, our analysis shows that our model is well understood and easy to apply, and it revealed the semantic properties of such review comments.

Digital Libraries
[The different models of scientific journals]

Jean-Philippe Chippaux

DOI: https://doi.org/10.48327/mtsi.v3i4.2023.454

2023-12-08

Abstract:Introduction: Scientific journals are the main source of scientific data, ensuring their registration, validation, distribution and archiving. With over 2.6 million scientific articles published each year, the turnover of scientific journals exceeds $25 billion annually. Five publishers share nearly half of this lucrative market. Scientists are the key players in the process, but other stakeholders have gradually been introduced, building various business models whose similarities and differences are described here. Concepts underlying scientific publication: Open access to scientific papers dates back to the scientific and technical revolution of the 17th century. However, its evolution has been considerably boosted by the development of the Internet and the recognition of science as "commons".Scientific integrity is under the control of research institutions to ensure the prevention of fraud and misconduct in the course of scientific production. Usually, the scientific integrity is questioned during the manuscript reviewing process which may result in identification of flaws. Models of scientific publications: In the historical model, readers pay for access to the document. Authors are not remunerated and renounce copyright on their articles to the publisher. The limits of the historical model became clear in the 90s, facing the cost of publishing, reduction in the number of subscribers, development of the Internet and willingness to improve manuscript evaluation.With the development of the Internet and the paradigm of open access, publishers proposed a new model in the 2000s, replacing the cost of access to articles for the reader with the payment of Article Processing Charges (APCs) paid by the author or its institution ("author pays" model). In this model, the content of the article can be freely reproduced and used, provided that the original author is credited. In addition to the evaluation of the manuscript which remains a critical factor, the cost of publication appears inequitable. However, all or part of the APCs may be waived, particularly for authors from low- and middle-income countries.For the past 15 years or so, publishers, learned societies and academic or research institutions (including libraries) have been seeking to publish reliable, open access manuscripts that respect scientific integrity while being affordable for the author.Predatory journals emerged in the late 2000s, taking advantage of the success of the authorpays model to capture APCs. Lacking a proper evaluation process resulting in poor-quality publications, these journals are rejected by most scientific institutions. On the other hand, they are particularly attractive in low- and middle-income countries because of their aggressive commercial practices (insistent invitation to submit a manuscript, low rejection rate, rapid publication, reduced APC, etc.).The purpose of each journal is to secure its economic model. This goes through ensuring its visibility, which is determined by the number of citations (online and social media citations) rather than the quality of the articles published. Peer review: This very old concept has not been widely used until the 20th century. In the historical model, manuscript evaluation is generally carried out by members of the learned society that publishes the journal. Evaluation can be either unblinded, single-blinded (referee is anonymous) or double-blinded (author and referee are anonymous). Several studies have shown that blind procedures do not alter the quality of the evaluation. Since the early 90s, post-publication evaluation has emerged, of which there are several variants. The aim is to shorten times to publication and open up the evaluation process more widely in order to limit the bias. Apart from the fact that this system does not guarantee a better evaluation of the manuscript, its main disadvantage is that the article is accessible without validation of the data collection and analysis throughout the entire process, which can be lengthy. Cost and funding of scientific journals: The cost of an article depends on charges that vary according to the conditions and places of production. Reviewers are volunteers. On the other hand, manuscript management, editing and distribution are carried out by professionals, which entail financial charges. Some of these costs are lower in low- and medium-resource countries, where economies of scale and higher benefits are possible. Conclusion: The limits of the historical model have led to the development of several business models of scientific journals, that are in constant evolution, especially the author pays model which promises open access to publications but impacts scientific production. However, the evaluation of scientific production is heterogeneous due to a limited pool of reviewers inadequately selected. Scientific publishing is looking for solutions to find a virtuous model that respects open science, open access to data and scientific integrity. The "Diamond open access" model, free of charge for both readers and authors with the guarantee of an irrevocable license to reproduce the content of the article provided that the original source is cited, perfectly fits.
The Miracle of Peer Review and Development in Science: An Agent-Based Model

Simone Righi,Károly Takács

DOI: https://doi.org/10.48550/arXiv.1607.02574

2016-07-09

Abstract:It is not easy to rationalize how peer review, as the current grassroots of science, can work based on voluntary contributions of reviewers. There is no rationale to write impartial and thorough evaluations. Consequently, there is no risk in submitting low-quality work by authors. As a result, scientists face a social dilemma: if everyone acts according to his or her own self-interest, low scientific quality is produced. Still, in practice, reviewers as well as authors invest high effort in reviews and submissions. We examine how the increased relevance of public good benefits (journal impact factor), the editorial policy of handling incoming reviews, and the acceptance decisions that take into account reputational information can help the evolution of high-quality contributions from authors. High effort from the side of reviewers is problematic even if authors cooperate: reviewers are still best off by producing low-quality reviews, which does not hinder scientific development, just adds random noise and unnecessary costs to it. We show with agent-based simulations that tacit agreements between authors that are based on reciprocity might decrease these costs, but does not result in superior scientific quality. Our study underlines why certain self-emerged current practices, such as the increased importance of journal metrics, the reputation-based selection of reviewers, and the reputation bias in acceptance work efficiently for scientific development. Our results find no answers, however, how the system of peer review with impartial and thorough evaluations could be sustainable jointly with rapid scientific development.

Physics and Society,Digital Libraries,Multiagent Systems
A conceptual peer review model for arXiv and other preprint databases

LingFeng Wang,YaQing Zhan

DOI: https://doi.org/10.1002/leap.1229

2019-02-06

Learned Publishing

Abstract:A global survey conducted by arXiv in 2016 showed that 58% of arXiv users thought arXiv should have a peer review system. The current opinion is that arXiv should adopt the Community Peer Review model. This paper evaluates and identifies two weak points of Community Peer Review and proposes a new peer review model – Self‐Organizing Peer Review. We propose a model in which automated methods of matching reviewers to articles and ranking both users and articles can be implemented. In addition, we suggest a strategic plan to increase recognition of articles in preprint databases within academic circles so that second generation preprint databases can achieve faster and cheaper publication.

information science & library science
Peer Review as an Evolving Response to Organizational Constraint: Evidence from Sociology Journals, 1952–2018

Ben Merriman

DOI: https://doi.org/10.1007/s12108-020-09473-x

2020-11-18

The American Sociologist

Abstract:Double-blind peer review is a central feature of the editorial model of most journals in sociology and neighboring social scientific fields, yet there is little history of how and when its main features developed. Drawing from nearly 70 years of annual reports of the editors of American Sociological Association journals, this article describes the historical emergence of major elements of editorial peer review. These reports and associated descriptive statistics are used to show that blind review, ad hoc review, the formal requirement of exclusive submission, routine use of the revise and resubmit decision, and common use of desk rejection developed separately over a period of decades. The article then argues that the ongoing evolution of the review model has not been driven by intellectual considerations. Rather, the evolution of peer review is best understood as the product of continuous efforts to steward editors' scarce attention while preserving an open submission policy that favors authors' interests.
Publish and Perish: Why the Current Publication and Review Model is Killing Research and Wasting Your Money.

Fabio Casati,Fausto Giunchiglia,Maurizio Marchese

DOI: https://doi.org/10.1145/1226694.1226695

2010-01-01

Abstract:The research world, and specifically the academic world, is centered around the notion of publication as the basic mean to disseminate results, foster interaction among communities, and achieve international recognition (and career advancement). Publications are done in conferences or journals, and are usually reviewed by a committee of experts, also referred as “peers”. Typically, each paper is reviewed by 3 or 4 reviewers. The “best” papers among all the submitted ones are then accepted for publication in the journal or in the conference proceedings. In the computer science area, people typically publishes a dozen paper per year, and submit a little more than that (not all papers are accepted the first time around). Acceptance rates for conferences are often around 20% or lower.
On novel peer review system for academic journals: analysis based on social computing

Li Liu,Qian Wang,Zong-Yuan Tan,Ning Cai

DOI: https://doi.org/10.1007/s11071-023-08401-1

IF: 5.741

2023-03-27

Nonlinear Dynamics

Abstract:For improving the performance and effectiveness of peer review, a novel review system is proposed, based on analysis of peer review process for academic journals under a parallel model built via Monte Carlo method. The model can simulate the review, application, and acceptance activities of the review systems, in a distributed manner. Simulation experiments are operated on two distinctive review systems. Significant advantages of the novel system are shown by the results.

engineering, mechanical,mechanics
The peer review game: an agent-based model of scientists facing resource constraints and institutional pressures

Federico Bianchi,Francisco Grimaldo,Giangiacomo Bravo,Flaminio Squazzoni

DOI: https://doi.org/10.1007/s11192-018-2825-4

IF: 3.801

2018-07-09

Scientometrics

Abstract:This paper looks at peer review as a cooperation dilemma through a game-theory framework. We built an agent-based model to estimate how much the quality of peer review is influenced by different resource allocation strategies followed by scientists dealing with multiple tasks, i.e., publishing and reviewing. We assumed that scientists were sensitive to acceptance or rejection of their manuscripts and the fairness of peer review to which they were exposed before reviewing. We also assumed that they could be realistic or excessively over-confident about the quality of their manuscripts when reviewing. Furthermore, we assumed they could be sensitive to competitive pressures provided by the institutional context in which they were embedded. Results showed that the bias and quality of publications greatly depend on reviewer motivations but also that context pressures can have a negative effect. However, while excessive competition can be detrimental to minimising publication bias, a certain level of competition is instrumental to ensure the high quality of publication especially when scientists accept reviewing for reciprocity motives.

information science & library science,computer science, interdisciplinary applications
The limitations to our understanding of peer review

Jonathan P. Tennant,Tony Ross-Hellauer

DOI: https://doi.org/10.1186/s41073-020-00092-1

2020-04-30

Research Integrity and Peer Review

Abstract:Abstract Peer review is embedded in the core of our knowledge generation systems, perceived as a method for establishing quality or scholarly legitimacy for research, while also often distributing academic prestige and standing on individuals. Despite its critical importance, it curiously remains poorly understood in a number of dimensions. In order to address this, we have analysed peer review to assess where the major gaps in our theoretical and empirical understanding of it lie. We identify core themes including editorial responsibility, the subjectivity and bias of reviewers, the function and quality of peer review, and the social and epistemic implications of peer review. The high-priority gaps are focused around increased accountability and justification in decision-making processes for editors and developing a deeper, empirical understanding of the social impact of peer review. Addressing this at the bare minimum will require the design of a consensus for a minimal set of standards for what constitutes peer review, and the development of a shared data infrastructure to support this. Such a field requires sustained funding and commitment from publishers and research funders, who both have a commitment to uphold the integrity of the published scholarly record. We use this to present a guide for the future of peer review, and the development of a new research discipline based on the study of peer review.
A history and development of peer-review process

Siladitya Jana

2019-07-12

Abstract:The paper shows the importance of peer review process in the scholarly communication system and discusses both the closed and the newly emerging open peer review models. It also examines the peer review system at the scholarly academies or societies in their nomination systems for prizes, rewards, etc. It also discusses the various facets of the newly developed open peer review models now prevalent in various journals. The paper may help to understand and appreciate the role played by peer review in the scholarly communication system and the efforts being made to make it more transparent.

History,Education,Political Science
Editorial Behaviors in Peer Review

Wei Wang,Xiangjie Kong,Jun Zhang,Zhen Chen,Feng Xia,Xianwen Wang

DOI: https://doi.org/10.1186/s40064-016-2601-y

2016-01-01

SpringerPlus

Abstract:Editors play a critical role in the peer review system. How do editorial behaviors affect the performance of peer review? No quantitative model to date allows us to measure the influence of editorial behaviors on different peer review stages such as, manuscript distribution and final decision making. Here, we propose an agent-based model in which the process of peer review is guided mainly by the social interactions among three kinds of agents representing authors, editors and reviewers respectively. We apply this model to analyze a number of editorial behaviors such as decision strategy, number of reviewers and editorial bias on peer review. We find out that peer review outcomes are significantly sensitive to different editorial behaviors. With a small fraction (10%) of biased editors, the quality of accepted papers declines 11%, which indicates that effects of editorial biased behavior is worse than that of biased reviewers (7%). While several peer review models exist, this is the first account for the study of editorial behaviors that is validated on the basis of simulation analysis.
Is Peer Review in Academic Publishing Still Working?

Liz Jackson,Michael A. Peters,Leon Benadé,Nesta Devine,Sonja Arndt,Daniella J. Forster,Andrew Gibbons,Elizabeth Grierson,Petar Jandrić,George Lăzăroiu,Kirsten Locke,Ramona Mihăilă,Georgina Stewart,Marek Tesař,Peter Roberts,Jānis Tālivaldis Ozoliņš

DOI: https://doi.org/10.1080/23265507.2018.1479139

2018-01-01

Open Review of Educational Research

Abstract:Peer review is central to academic publishing. Yet for many it is a mysterious and contentious practice, which can cause distress for both reviewers, and those whose work is reviewed. This paper, produced by the Editors' Collective, examines the past and future of peer review in academic publishing. The first sections consider how peer review has been defined and practised in changing academic contexts, and its educational significance in the development of scholarship. The paper then explores major historical and contemporary issues around identity, diversity, anonymity, and the review process, and the related power of editors versus reviewers in academic publishing. Finally, the paper discusses the case of new scholars as reviewers engaging in neoliberal labour, before concluding with some brief recommendations based on our analysis.
Academics evaluating academics: a methodology to inform the review process on top of open citations

Federica Bologna,Angelo Di Iorio,Silvio Peroni,Francesco Poggi

DOI: https://doi.org/10.48550/arXiv.2106.05725

2021-06-10

Digital Libraries

Abstract:In the past, several works have investigated ways for combining quantitative and qualitative methods in research assessment exercises. In this work, we aim at introducing a methodology to explore whether citation-based metrics, calculated only considering open bibliographic and citation data, can yield insights on how human peer-review of research assessment exercises is conducted. To understand if and what metrics provide relevant information, we propose to use a series of machine learning models to replicate the decisions of the committees of the research assessment exercises.

Peer review and journal models

Evaluating alternative systems of peer review: a large-scale agent-based modelling approach to scientific publication

Complex systems approach to scientific publication and peer-review system: development of an agent-based model calibrated with empirical journal data

A multi-disciplinary perspective on emergent and future innovations in peer review

Accelerating the pace of discovery by changing the peer review algorithm

Peer-review in the Internet age

Peer review vs bibliometrics: which method better predicts the scholarly impact of publications?

The Peer Review Process: Past, Present, and Future

Peer Reviewing Revisited: Assessing Research with Interlinked Semantic Comments

[The different models of scientific journals]

The Miracle of Peer Review and Development in Science: An Agent-Based Model

A conceptual peer review model for arXiv and other preprint databases

Peer Review as an Evolving Response to Organizational Constraint: Evidence from Sociology Journals, 1952–2018

Publish and Perish: Why the Current Publication and Review Model is Killing Research and Wasting Your Money.

On novel peer review system for academic journals: analysis based on social computing

The peer review game: an agent-based model of scientists facing resource constraints and institutional pressures

The limitations to our understanding of peer review

A history and development of peer-review process

Editorial Behaviors in Peer Review

Is Peer Review in Academic Publishing Still Working?

Academics evaluating academics: a methodology to inform the review process on top of open citations