Abstract:Abstract Background The emergence of systems based on large language models (LLMs) such as OpenAI’s ChatGPT has created a range of discussions in scholarly circles. Since LLMs generate grammatically correct and mostly relevant (yet sometimes outright wrong, irrelevant or biased) outputs in response to provided prompts, using them in various writing tasks including writing peer review reports could result in improved productivity. Given the significance of peer reviews in the existing scholarly publication landscape, exploring challenges and opportunities of using LLMs in peer review seems urgent. After the generation of the first scholarly outputs with LLMs, we anticipate that peer review reports too would be generated with the help of these systems. However, there are currently no guidelines on how these systems should be used in review tasks. Methods To investigate the potential impact of using LLMs on the peer review process, we used five core themes within discussions about peer review suggested by Tennant and Ross-Hellauer. These include 1) reviewers’ role, 2) editors’ role, 3) functions and quality of peer reviews, 4) reproducibility, and 5) the social and epistemic functions of peer reviews. We provide a small-scale exploration of ChatGPT’s performance regarding identified issues. Results LLMs have the potential to substantially alter the role of both peer reviewers and editors. Through supporting both actors in efficiently writing constructive reports or decision letters, LLMs can facilitate higher quality review and address issues of review shortage. However, the fundamental opacity of LLMs’ training data, inner workings, data handling, and development processes raise concerns about potential biases, confidentiality and the reproducibility of review reports. Additionally, as editorial work has a prominent function in defining and shaping epistemic communities, as well as negotiating normative frameworks within such communities, partly outsourcing this work to LLMs might have unforeseen consequences for social and epistemic relations within academia. Regarding performance, we identified major enhancements in a short period and expect LLMs to continue developing. Conclusions We believe that LLMs are likely to have a profound impact on academia and scholarly communication. While potentially beneficial to the scholarly communication system, many uncertainties remain and their use is not without risks. In particular, concerns about the amplification of existing biases and inequalities in access to appropriate infrastructure warrant further attention. For the moment, we recommend that if LLMs are used to write scholarly reviews and decision letters, reviewers and editors should disclose their use and accept full responsibility for data security and confidentiality, and their reports’ accuracy, tone, reasoning and originality.

Prompting LLMs to Compose Meta-Review Drafts from Peer-Review Narratives of Scholarly Manuscripts

LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewing

ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewing

Evoke: Evoking Critical Thinking Abilities in LLMs via Reviewer-Author Prompt Editing

Reviewer2: Optimizing Review Generation Through Prompt Generation

LLMs for Literature Review: Are we there yet?

AI-Driven Review Systems: Evaluating LLMs in Scalable and Bias-Aware Academic Reviews

A Sentiment Consolidation Framework for Meta-Review Generation

The emergence of Large Language Models (LLM) as a tool in literature reviews: an LLM automated systematic review

Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review

Prompting and Fine-tuning Large Language Models for Automated Code Review Comment Generation

Peer Review as A Multi-Turn and Long-Context Dialogue with Role-Based Interactions

Automatic Large Language Model Evaluation Via Peer Review

Review-LLM: Harnessing Large Language Models for Personalized Review Generation

LitLLM: A Toolkit for Scientific Literature Review

Generative Adversarial Reviews: When LLMs Become the Critic

Fine-tuning and prompt engineering for large language models-based code review automation

Cutting Through the Clutter: The Potential of LLMs for Efficient Filtration in Systematic Literature Reviews

One Prompt To Rule Them All: LLMs for Opinion Summary Evaluation

The Promise and Challenges of Using LLMs to Accelerate the Screening Process of Systematic Reviews

Prompting is all you need: LLMs for systematic review screening