Abstract:This research to practice full paper presented a peer review approach to grading projects in computer courses. Educators commonly adopt project-based learning activities in computer courses to achieve the goal. Implementations of these projects typically involve hardly quantifiable dimensions, such as novelty, functionality, user-friendly, coding style, and document. Automated tools are not competent for project assessment. However, assessing projects by instructors can be labor intensive and generally involves a high degree of subjectivity. Peer review is widely used in high education as effective active learning in computer courses, especially in the past decade. Despite its many advantages, peer review still faces some challenges. Students need to be more expert in assessment and might make mistakes when grading, and instructors still need to make great efforts to supervise the process and ensure that students give fair scores to their peers. In addition, the social relationship between students could lead to subjective assessments and affect the fairness of grading. We present the peer review approach to grading projects in computer courses. First, we design a grading specification to improve the accuracy of scoring. Second, we propose a score calculation method based on Technique for Order Preference by Similarity to an Ideal Solution (TOPSIS) to calculate the assignment's final score, which is a multi-objective decision-making problem. Third, we give an anomaly detection algorithm based on Dixon's Q test to filter peer reviewers' unreasonable scores. Based on the approach, we designed a peer review workflow and developed an online peer review tool called Peer Review Studio (PRS). We carry out the method in the undergraduate computer course of 2022. We collect and analyze the learning data between 2020 and 2022. When measured by Krippendorff's alpha, the inter-rater reliability between instructor and peer grading is in good agreement. When measured by Kruskal-Wallis, students' project performance is significantly improved in the first year of peer review. Although learning engagement increases in 2022, there is no statistically significant difference with 2021. The course questionnaire of 2022 shows that most students approve of the peer review approach.

An Empirical Implementation of Peer Review in Software Development

Comparative Case Studies of Open Source Software Peer Review Practices

Recommending Code Reviewers for Proprietary Software Projects: A Large Scale Study

An Evolutionary Game of Partners in Peer Review

Behind Linus's law: A preliminary analysis of open source software peer review practices in Mozilla and Python

Distributed Software Engineering Management

P2P-Based software engineering management

Code Reviewer Recommendation in Tencent: Practice, Challenge, and Direction*

Code Reviewer Recommendation in Tencent

An Empirically-Based Process To Improve The Practice Of Requirement Review

Process Improvement Of Peer Code Review And Behavior Analysis Of Its Participants

Quality Assurance of Peer Code Review Process: A Computer Science Based Strategy

The Status Quo of Peer Review Research and Its Problems

Software Review Procedures

Developers perception of peer code review in research software development

A customizable agile software Quality Assurance model

Understanding Peer Review of Software Engineering Papers

Integrating PSP with Agile Process:A Systematic Review

Quality Assurance of Peer Code Review Process: A Web-Based MIS

A Peer Review Approach to Grading Projects in Computer Courses.