Characterizing Peer Review Comments of Academic Articles in Multiple Rounds

Han Ruxue,Zhou Haomin,Zhong Jiangtao,Zhang Chengzhi

DOI: https://doi.org/10.1002/pra2.607

2022-10-15

Proceedings of the American Society for Information Science and Technology

Abstract:Proceedings of the Association for Information Science and Technology, Volume 59, Issue 1, Page 89-99, 2022.

What problem does this paper attempt to address?

Four steps to realize effective peer review of academics journals published in China

Fang PENG,Jianhua JIN,Jianrong HUANG

DOI: https://doi.org/10.11946/cjstp.201612191002

2017-01-01

Abstract:[Purposes] This paper aims to explore the strategies to achieve the effective peer review.[Methods] Based on advanced experience and measures taken by foreign journals,the problems existing in peer review of academics journals published in China were analyzed and improvement measures from the top level design were put forward.[Findings] Concrete measures include policy support and protection,comprehensive evaluation experts teams,optimized review patterns and sound evaluation and feedback systems.[Conclusions] Four kinds of measures mentioned above contribute to implementing the effective role of peer review in academic supervision and checks.
Peer Reviewing Revisited: Assessing Research with Interlinked Semantic Comments

Cristina-Iulia Bucur,Tobias Kuhn,Davide Ceolin

DOI: https://doi.org/10.48550/arXiv.1910.03218

2019-10-08

Abstract:Scientific publishing seems to be at a turning point. Its paradigm has stayed basically the same for 300 years but is now challenged by the increasing volume of articles that makes it very hard for scientists to stay up to date in their respective fields. In fact, many have pointed out serious flaws of current scientific publishing practices, including the lack of accuracy and efficiency of the reviewing process. To address some of these problems, we apply here the general principles of the Web and the Semantic Web to scientific publishing, focusing on the reviewing process. We want to determine if a fine-grained model of the scientific publishing workflow can help us make the reviewing processes better organized and more accurate, by ensuring that review comments are created with formal links and semantics from the start. Our contributions include a novel model called Linkflows that allows for such detailed and semantically rich representations of reviews and the reviewing processes. We evaluate our approach on a manually curated dataset from several recent Computer Science journals and conferences that come with open peer reviews. We gathered ground-truth data by contacting the original reviewers and asking them to categorize their own review comments according to our model. Comparing this ground truth to answers provided by model experts, peers, and automated techniques confirms that our approach of formally capturing the reviewers' intentions from the start prevents substantial discrepancies compared to when this information is later extracted from the plain-text comments. In general, our analysis shows that our model is well understood and easy to apply, and it revealed the semantic properties of such review comments.

Digital Libraries
Peer reviewers equally critique theory, method, and writing, with limited effect on the final content of accepted manuscripts

Dimity Stephen

DOI: https://doi.org/10.1007/s11192-022-04357-y

IF: 3.801

Scientometrics

Abstract:The primary aims of peer review are to detect flaws and deficiencies in the design and interpretation of studies, and ensure the clarity and quality of their presentation. However, it has been questioned whether peer review fulfils this function. Studies have highlighted a stronger focus of reviewers on critiquing methodological aspects of studies and the quality of writing in biomedical sciences, with less focus on theoretical grounding. In contrast, reviewers in the social sciences appear more concerned with theoretical underpinnings. These studies also found the effect of peer review on manuscripts' content to be variable, but generally modest and positive. I qualitatively analysed 1430 peer reviewers' comments for a sample of 40 social science preprint-publication pairs to identify the key foci of reviewers' comments. I then quantified the effect of peer review on manuscripts by examining differences between the preprint and published versions using the normalised Levenshtein distance, cosine similarity, and word count ratios for titles, abstracts, document sections and full-texts. I also examined changes in references used between versions and linked changes to reviewers' comments. Reviewers' comments were nearly equally split between issues of methodology (30.7%), theory (30.0%), and writing quality (29.2%). Titles, abstracts, and the semantic content of documents remained similar, although publications were typically longer than preprints. Two-thirds of citations were unchanged, 20.9% were added during review and 13.1% were removed. These findings indicate reviewers equally attended to the theoretical and methodological details and communication style of manuscripts, although the effect on quantitative measures of the manuscripts was limited. Supplementary information: The online version contains supplementary material available at 10.1007/s11192-022-04357-y.
Peer reviewers equally critique theory, method, and writing, with limited effect on manuscripts' content

Dimity Stephen

DOI: https://doi.org/10.48550/arXiv.2110.07950

2021-10-15

Abstract:Peer review aims to detect flaws and deficiencies in the design and interpretation of studies, and ensure the clarity and quality of their presentation. However, it has been questioned whether peer review fulfils this function. Studies have highlighted a stronger focus of reviewers on critiquing methodological aspects of studies and the quality of writing in biomedical sciences, with less focus on theoretical grounding. In contrast, reviewers in the social sciences appear more concerned with theoretical underpinnings. These studies also found the effect of peer review on manuscripts' content to be variable, but generally modest and positive. I qualitatively analysed 1,430 peer reviewers' comments for a sample of 40 social science preprint-publication pairs to identify the key foci of reviewers' comments. I then quantified the effect of peer review on manuscripts by examining differences between the preprint and published versions using the normalised Levenshtein distance, cosine similarity, and word count ratios for titles, abstracts, document sections and full-texts. I also examined changes in references used between versions and linked changes to reviewers' comments. Reviewers' comments were nearly equally split between issues of methodology (30.7%), theory (30.0%), and writing quality (29.2%). Titles, abstracts, and the semantic content of documents remained similar, although publications were typically longer than preprints. Two-thirds of citations were unchanged, 20.9% were added during review and 13.1% were removed. These findings indicate reviewers equally attended to the theoretical and methodological details and communication style of manuscripts, although the effect on quantitative measures of the manuscripts was limited.

Digital Libraries
Understanding the peer review endeavor in scientific publishing

Guangyao Zhang,Shenmeng Xu,Yao Sun,Chunlin Jiang,Xianwen Wang

DOI: https://doi.org/10.1016/j.joi.2022.101264

IF: 3.7

2022-05-01

Journal of Informetrics

Abstract:Peer review plays an essential role in the scholarly publishing life cycle. Using the verified peer review records of reviewers who use the Publons, we employed review length as a potential indicator of the effort researchers spend on peer review. We then examined the associations between various factors and review length. Special focus was placed on estimating the relationships between non-academic (economic and sociological aspects) factors and review length. Our results show that gender, country-level cultural backgrounds, and country-level economic backgrounds were significantly associated with review length. In addition, there are significant associations of disciplines (humanities & social sciences or hard sciences), English proficiency, publications, and verified reviews with review length.

information science & library science,computer science, interdisciplinary applications
Advantages and feasibility of double-blind peer-review for research articles

ZHENG Meiling,YANG Lili,SHU Qiang,LI Minju

DOI: https://doi.org/10.16811/j.cnki.1001-4314.2008.06.039

2008-01-01

Abstract:The merits of the double-blind form over the other two forms are emphasized in this paper. Single-blind peer-review is unbalanced in form, and is apt to result in lack of accountability of the referees and unfair judgment caused by personal bias. Open peer review is transparent, but it has not been mature enough to be widely adopted, especially in the present China. Double-blind peer-review is more objective and fairer compared with the single-blind form. It is favored by most researchers, and has the potential of improving the review quality and journal citation. Editors should fully recognize the double-blind form so as to play active and effective roles in the review process.
What Can Natural Language Processing Do for Peer Review?

Ilia Kuznetsov,Osama Mohammed Afzal,Koen Dercksen,Nils Dycke,Alexander Goldberg,Tom Hope,Dirk Hovy,Jonathan K. Kummerfeld,Anne Lauscher,Kevin Leyton-Brown,Sheng Lu,Mausam,Margot Mieskes,Aurélie Névéol,Danish Pruthi,Lizhen Qu,Roy Schwartz,Noah A. Smith,Thamar Solorio,Jingyan Wang,Xiaodan Zhu,Anna Rogers,Nihar B. Shah,Iryna Gurevych

2024-05-11

Abstract:The number of scientific articles produced every year is growing rapidly. Providing quality control over them is crucial for scientists and, ultimately, for the public good. In modern science, this process is largely delegated to peer review -- a distributed procedure in which each submission is evaluated by several independent experts in the field. Peer review is widely used, yet it is hard, time-consuming, and prone to error. Since the artifacts involved in peer review -- manuscripts, reviews, discussions -- are largely text-based, Natural Language Processing has great potential to improve reviewing. As the emergence of large language models (LLMs) has enabled NLP assistance for many new tasks, the discussion on machine-assisted peer review is picking up the pace. Yet, where exactly is help needed, where can NLP help, and where should it stand aside? The goal of our paper is to provide a foundation for the future efforts in NLP for peer-reviewing assistance. We discuss peer review as a general process, exemplified by reviewing at AI conferences. We detail each step of the process from manuscript submission to camera-ready revision, and discuss the associated challenges and opportunities for NLP assistance, illustrated by existing work. We then turn to the big challenges in NLP for peer review as a whole, including data acquisition and licensing, operationalization and experimentation, and ethical issues. To help consolidate community efforts, we create a companion repository that aggregates key datasets pertaining to peer review. Finally, we issue a detailed call for action for the scientific community, NLP and AI researchers, policymakers, and funding bodies to help bring the research in NLP for peer review forward. We hope that our work will help set the agenda for research in machine-assisted scientific quality control in the age of AI, within the NLP community and beyond.

Computation and Language
Automated Peer Reviewing in Paper SEA: Standardization, Evaluation, and Analysis

Jianxiang Yu,Zichen Ding,Jiaqi Tan,Kangyang Luo,Zhenmin Weng,Chenghua Gong,Long Zeng,Renjing Cui,Chengcheng Han,Qiushi Sun,Zhiyong Wu,Yunshi Lan,Xiang Li

2024-10-02

Abstract:In recent years, the rapid increase in scientific papers has overwhelmed traditional review mechanisms, resulting in varying quality of publications. Although existing methods have explored the capabilities of Large Language Models (LLMs) for automated scientific reviewing, their generated contents are often generic or partial. To address the issues above, we introduce an automated paper reviewing framework SEA. It comprises of three modules: Standardization, Evaluation, and Analysis, which are represented by models SEA-S, SEA-E, and SEA-A, respectively. Initially, SEA-S distills data standardization capabilities of GPT-4 for integrating multiple reviews for a paper. Then, SEA-E utilizes standardized data for fine-tuning, enabling it to generate constructive reviews. Finally, SEA-A introduces a new evaluation metric called mismatch score to assess the consistency between paper contents and reviews. Moreover, we design a self-correction strategy to enhance the consistency. Extensive experimental results on datasets collected from eight venues show that SEA can generate valuable insights for authors to improve their papers.

Computation and Language,Digital Libraries,Information Retrieval
Evaluating the Pros and Cons of Different Peer Review Policies Via Simulation

Jia Zhu,Gabriel Fung,Wai Hung Wong,Zhixu Li,Chuanhua Xu

DOI: https://doi.org/10.1007/s11948-015-9683-8

IF: 3.777

2015-01-01

Science and Engineering Ethics

Abstract:In the academic world, peer review is one of the major processes in evaluating a scholars contribution. In this study, we are interested in quantifying the merits of different policies in a peer review process, such as single-blind review, double-blind review, and obtaining authors feedback. Currently, insufficient work has been undertaken to evaluate the benefits of different peer review policies. One of the major reasons for this situation is the inability to conduct any empirical study because data are presently unavailable. In this case, a computer simulation is one of the best ways to conduct a study. We perform a series of simulations to study the effects of different policies on a peer review process. In this study, we focus on the peer review process of a typical computer science conference. Our results point to the crucial role of program chairs in determining the quality and diversity of the articles to be accepted for publication. We demonstrate the importance of discussion among reviewers, suggest circumstances in which the double-blind review policy should be adopted, and question the credibility of the authors feedback mechanism. Finally, we stress that randomness plays an important role in the peer review process, and this role cannot be eliminated. Although our model may not capture every component of a peer review process, it covers some of the most essential elements. Thus, even the simulation results clearly cannot be taken as literal descriptions of an actual peer review process. However, we can at least still use them to identify alternative directions for future study.
AgentReview: Exploring Peer Review Dynamics with LLM Agents

Yiqiao Jin,Qinlin Zhao,Yiyang Wang,Hao Chen,Kaijie Zhu,Yijia Xiao,Jindong Wang

2024-10-13

Abstract:Peer review is fundamental to the integrity and advancement of scientific publication. Traditional methods of peer review analyses often rely on exploration and statistics of existing peer review data, which do not adequately address the multivariate nature of the process, account for the latent variables, and are further constrained by privacy concerns due to the sensitive nature of the data. We introduce AgentReview, the first large language model (LLM) based peer review simulation framework, which effectively disentangles the impacts of multiple latent factors and addresses the privacy issue. Our study reveals significant insights, including a notable 37.1% variation in paper decisions due to reviewers' biases, supported by sociological theories such as the social influence theory, altruism fatigue, and authority bias. We believe that this study could offer valuable insights to improve the design of peer review mechanisms. Our code is available at <a class="link-external link-https" href="https://github.com/Ahren09/AgentReview" rel="external noopener nofollow">this https URL</a>.

Computation and Language
Classification and analysis of PubPeer comments: How a web journal club is used

José Luis Ortega

DOI: https://doi.org/10.1002/asi.24568

2021-08-24

Journal of the Association for Information Science and Technology

Abstract:This study explores the use of PubPeer by the scholarly community, to understand the issues discussed in an online journal club, the disciplines most commented on, and the characteristics of the most prolific users. A sample of 39,985 posts about 24,779 publications were extracted from PubPeer in 2019 and 2020. These comments were divided into seven categories according to their degree of seriousness (Positive review, Critical review, Lack of information, Honest errors, Methodological flaws, Publishing fraud, and Manipulation). The results show that more than two-thirds of comments are posted to report some type of misconduct, mainly about image manipulation. These comments generate most discussion and take longer to be posted. By discipline, Health Sciences and Life Sciences are the most discussed research areas. The results also reveal "super commenters," users who access the platform to systematically review publications. The study ends by discussing how various disciplines use the site for different purposes.

information science & library science,computer science, information systems
Peer review analyze: A novel benchmark resource for computational analysis of peer reviews

Tirthankar Ghosal,Sandeep Kumar,Prabhat Kumar Bharti,Asif Ekbal

DOI: https://doi.org/10.1371/journal.pone.0259238

IF: 3.7

2022-01-27

PLoS ONE

Abstract:Peer Review is at the heart of scholarly communications and the cornerstone of scientific publishing. However, academia often criticizes the peer review system as non-transparent, biased, arbitrary, a flawed process at the heart of science, leading to researchers arguing with its reliability and quality. These problems could also be due to the lack of studies with the peer-review texts for various proprietary and confidentiality clauses. Peer review texts could serve as a rich source of Natural Language Processing (NLP) research on understanding the scholarly communication landscape, and thereby build systems towards mitigating those pertinent problems. In this work, we present a first of its kind multi-layered dataset of 1199 open peer review texts manually annotated at the sentence level (∼ 17k sentences) across the four layers, viz. Paper Section Correspondence, Paper Aspect Category, Review Functionality, and Review Significance. Given a text written by the reviewer, we annotate: to which sections (e.g., Methodology, Experiments, etc.), what aspects (e.g., Originality/Novelty, Empirical/Theoretical Soundness, etc.) of the paper does the review text correspond to, what is the role played by the review text (e.g., appreciation, criticism, summary, etc.), and the importance of the review statement (major, minor, general) within the review. We also annotate the sentiment of the reviewer (positive, negative, neutral) for the first two layers to judge the reviewer's perspective on the different sections and aspects of the paper. We further introduce four novel tasks with this dataset, which could serve as an indicator of the exhaustiveness of a peer review and can be a step towards the automatic judgment of review quality. We also present baseline experiments and results for the different tasks for further investigations. We believe our dataset would provide a benchmark experimental testbed for automated systems to leverage on current NLP state-of-the-art techniques to address different issues with peer review quality, thereby ushering increased transparency and trust on the holy grail of scientific research validation. Our dataset and associated codes are available at https://www.iitp.ac.in/~ai-nlp-ml/resources.html#Peer-Review-Analyze.
A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications

Dongyeop Kang,Waleed Ammar,Bhavana Dalvi,Madeleine van Zuylen,Sebastian Kohlmeier,Eduard Hovy,Roy Schwartz

DOI: https://doi.org/10.48550/arXiv.1804.09635

2018-04-25

Computation and Language

Abstract:Peer reviewing is a central component in the scientific publishing process. We present the first public dataset of scientific peer reviews available for research purposes (PeerRead v1) providing an opportunity to study this important artifact. The dataset consists of 14.7K paper drafts and the corresponding accept/reject decisions in top-tier venues including ACL, NIPS and ICLR. The dataset also includes 10.7K textual peer reviews written by experts for a subset of the papers. We describe the data collection process and report interesting observed phenomena in the peer reviews. We also propose two novel NLP tasks based on this dataset and provide simple baseline models. In the first task, we show that simple models can predict whether a paper is accepted with up to 21% error reduction compared to the majority baseline. In the second task, we predict the numerical scores of review aspects and show that simple models can outperform the mean baseline for aspects with high variance such as 'originality' and 'impact'.
Scientific Commentaries Are Dealing with Uncertainty and Complexity in Science

Shuang Wang,Xin He,Jian Du

DOI: https://doi.org/10.1016/j.ipm.2024.103707

IF: 7.466

2024-01-01

Information Processing & Management

Abstract:Authors tend to cite scientific knowledge with higher certainty and lower complexity. Scientific comments, such as editorials and letters to the editor, may signify a distinction by offering constructive compliments or criticism on prior work. This study aims to understand why certain papers receive formal comments and their applications. We find that the rate of commented papers has remained stable at around 4 % for 30 years. Next, to explore the characteristics attracting formal comments, we designed a comparative study between over 0.8 million commented biomedical papers (study group) and the entire over 30 million PubMed records (baseline control group). First, based on citation sentiments of over 17.7 million documents from Scite.ai, we find that commented papers tend to be highly cited, whereas strongly contradicted and intensely disputing (i.e., receive both positive and contradicted citations), with a rate ratio of 2.2 times of baseline. This reflected the inherent controversy of content in triggering extensive argumentative discussion. Second, by mapping descriptors and qualifiers of Medical Subject Headings between the commented group and the baseline group, we observe that the prominence level for commented papers ranked from the social-related to biological-related fields and from applied to basic research topics. The observed spectrum aligns with the “hierarchy of sciences” model, in which the “hardness” of science decreases, and its complexity increases. Commented papers and their accompanying comments may provide a corpus for argument mining on uncertain and complex topics to support more informed and unbiased decision-making.
Peer Review as A Multi-Turn and Long-Context Dialogue with Role-Based Interactions

Cheng Tan,Dongxin Lyu,Siyuan Li,Zhangyang Gao,Jingxuan Wei,Siqi Ma,Zicheng Liu,Stan Z. Li

2024-06-09

Abstract:Large Language Models (LLMs) have demonstrated wide-ranging applications across various fields and have shown significant potential in the academic peer-review process. However, existing applications are primarily limited to static review generation based on submitted papers, which fail to capture the dynamic and iterative nature of real-world peer reviews. In this paper, we reformulate the peer-review process as a multi-turn, long-context dialogue, incorporating distinct roles for authors, reviewers, and decision makers. We construct a comprehensive dataset containing over 26,841 papers with 92,017 reviews collected from multiple sources, including the top-tier conference and prestigious journal. This dataset is meticulously designed to facilitate the applications of LLMs for multi-turn dialogues, effectively simulating the complete peer-review process. Furthermore, we propose a series of metrics to evaluate the performance of LLMs for each role under this reformulated peer-review setting, ensuring fair and comprehensive evaluations. We believe this work provides a promising perspective on enhancing the LLM-driven peer-review process by incorporating dynamic, role-based interactions. It aligns closely with the iterative and interactive nature of real-world academic peer review, offering a robust foundation for future research and development in this area. We open-source the dataset at <a class="link-external link-https" href="https://github.com/chengtan9907/ReviewMT" rel="external noopener nofollow">this https URL</a>.

Computation and Language,Artificial Intelligence,Machine Learning
Online peer editing: effects of comments and edits on academic writing skills

Han Zhang,Galina Shulgina,Mik Fanguy,Jamie Costley

DOI: https://doi.org/10.1016/j.heliyon.2022.e09822

IF: 3.776

2022-06-28

Heliyon

Abstract:Although the effects of online peer editing have been studied from a number of perspectives, it remains unclear how giving and receiving comments and edits affect student academic writing performance. The current study examined the influence of these aspects of peer editing on student academic writing performance in higher education during online peer editing. Participants were 76 students engaged in peer editing of one another's work in a graduate scientific writing course at a Korean university. The relationships between the giving and receiving of comments and edits, and student performance on their writing tasks were analyzed. Results showed that there is a positive correlation between the number of comments received and the student's writing score, whereas receiving edits had the opposite effect and was associated with lower student performance. Furthermore, no relationship was found between giving comments or edits and writing performance. These results add to the field's understanding of how specific elements of peer editing can impact students' performance.
On novel peer review system for academic journals: analysis based on social computing

Li Liu,Qian Wang,Zong-Yuan Tan,Ning Cai

DOI: https://doi.org/10.1007/s11071-023-08401-1

IF: 5.741

2023-03-27

Nonlinear Dynamics

Abstract:For improving the performance and effectiveness of peer review, a novel review system is proposed, based on analysis of peer review process for academic journals under a parallel model built via Monte Carlo method. The model can simulate the review, application, and acceptance activities of the review systems, in a distributed manner. Simulation experiments are operated on two distinctive review systems. Significant advantages of the novel system are shown by the results.

engineering, mechanical,mechanics
Can We Automate Scientific Reviewing?

Weizhe Yuan,Pengfei Liu,Graham Neubig

DOI: https://doi.org/10.1613/jair.1.12862

2022-09-29

Journal of Artificial Intelligence Research

Abstract:The rapid development of science and technology has been accompanied by an exponential growth in peer-reviewed scientific publications. At the same time, the review of each paper is a laborious process that must be carried out by subject matter experts. Thus, providing high-quality reviews of this growing number of papers is a significant challenge. In this work, we ask the question “can we automate scientific reviewing? ”, discussing the possibility of using natural language processing (NLP) models to generate peer reviews for scientific papers. Because it is non-trivial to define what a “good” review is in the first place, we first discuss possible evaluation metrics that could be used to judge success in this task. We then focus on the machine learning domain and collect a dataset of papers in the domain, annotate them with different aspects of content covered in each review, and train targeted summarization models that take in papers as input and generate reviews as output. Comprehensive experimental results on the test set show that while system-generated reviews are comprehensive, touching upon more aspects of the paper than human-written reviews, the generated texts are less constructive and less factual than human-written reviews for all aspects except the explanation of the core ideas of the papers, which are largely factually correct. Given these results, we pose eight challenges in the pursuit of a good review generation system together with potential solutions, which, hopefully, will inspire more future research in this direction. We make relevant resource publicly available for use by future research: https://github. com/neulab/ReviewAdvisor. In addition, while our conclusion is that the technology is not yet ready for use in high-stakes review settings we provide a system demo, ReviewAdvisor (http://review.nlpedia.ai/), showing the current capabilities and failings of state-of-the-art NLP models at this task (see demo screenshot in A.2). A review of this paper written by the system proposed in this paper can be found in A.1.

computer science, artificial intelligence
PeerBERT: Automated Characterization of Peer Review Comments across Courses

Amber Dood,Blair Winograd,Solaire Finkenstaedt-Quinn,Anne Gere,Ginger Shultz

DOI: https://doi.org/10.1145/3506860.3506892

2022-03-21

Abstract:Writing-to-learn pedagogies are an evidence-based practice known to aid students in constructing knowledge. Barriers exist for the implementation of such assignments; namely, instructors feel they do not have time to provide each student with feedback. To ease implementation of writing-to-learn assignments at scale, we have incorporated automated peer review, which facilitates peer review without input from the instructor. Participating in peer review can positively impact students’ learning and allow students to receive feedback on their writing. Instructors may want to monitor these peer interactions and gain insight into their students’ understanding using the feedback generated by their peers. To facilitate instructors’ use of the content from students’ peer review comments, we pre-trained a transformer model called PeerBERT. PeerBERT was fine-tuned on several downstream tasks to categorize students’ peer review comments as praise, problem/solution, or verification/summary. The model exhibits high accuracy, even across different peer review prompts, assignments, and courses. Additional downstream tasks label problem/solution peer review comments as one or more types: writing/formatting, missing content/needs elaboration, and incorrect content. This approach can help instructors pinpoint common issues in student writing by parsing out which comments are problem/solution and which type of problem/solution students identify.
How Did Reviewers Give Their Reviews? Characterizing Review Activity in Academic Conferences

Zhiwen Yu,Xinxin Zhang,Yunji Liang,Daqing Zhang

DOI: https://doi.org/10.14257/ijmue.2013.8.5.36

2013-01-01

International Journal of Multimedia and Ubiquitous Engineering

Abstract:Human activities have been investigated and applied in various fields, such as contextaware computing, search engine, social network services, location-based services, automated visual surveillance, and multimodal human-computer interaction.However, the human activities in the review process of academic conferences have seldom been explored.There is no doubt that review process plays an important role in deciding whether a paper can be accepted or not.In this paper, we present our work to understand the review activities by analyzing the anonymized review data of two conferences (ACM SIGCOMM and UIC).The descriptive statistics and the data mining technology are adopted in the analysis.We got some interesting knowledge, which is significant for interpreting how the reviewers give their reviews in academic conferences, such as the relationships between the score, confidence and review length, and reviewer activity patterns.

Characterizing Peer Review Comments of Academic Articles in Multiple Rounds

Four steps to realize effective peer review of academics journals published in China

Peer Reviewing Revisited: Assessing Research with Interlinked Semantic Comments

Peer reviewers equally critique theory, method, and writing, with limited effect on the final content of accepted manuscripts

Peer reviewers equally critique theory, method, and writing, with limited effect on manuscripts' content

Understanding the peer review endeavor in scientific publishing

Advantages and feasibility of double-blind peer-review for research articles

What Can Natural Language Processing Do for Peer Review?

Automated Peer Reviewing in Paper SEA: Standardization, Evaluation, and Analysis

Evaluating the Pros and Cons of Different Peer Review Policies Via Simulation

AgentReview: Exploring Peer Review Dynamics with LLM Agents

Classification and analysis of PubPeer comments: How a web journal club is used

Peer review analyze: A novel benchmark resource for computational analysis of peer reviews

A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications

Scientific Commentaries Are Dealing with Uncertainty and Complexity in Science

Peer Review as A Multi-Turn and Long-Context Dialogue with Role-Based Interactions

Online peer editing: effects of comments and edits on academic writing skills

On novel peer review system for academic journals: analysis based on social computing

Can We Automate Scientific Reviewing?

PeerBERT: Automated Characterization of Peer Review Comments across Courses

How Did Reviewers Give Their Reviews? Characterizing Review Activity in Academic Conferences