A Peer Review Approach to Grading Projects in Computer Courses.
Liang Zhang,Yue Zong,Tianyi Chen,Yafeng Ma,Shuo An,Lijun Zhang
DOI: https://doi.org/10.1109/FIE58773.2023.10343176
2023-01-01
Abstract:This research to practice full paper presented a peer review approach to grading projects in computer courses. Educators commonly adopt project-based learning activities in computer courses to achieve the goal. Implementations of these projects typically involve hardly quantifiable dimensions, such as novelty, functionality, user-friendly, coding style, and document. Automated tools are not competent for project assessment. However, assessing projects by instructors can be labor intensive and generally involves a high degree of subjectivity. Peer review is widely used in high education as effective active learning in computer courses, especially in the past decade. Despite its many advantages, peer review still faces some challenges. Students need to be more expert in assessment and might make mistakes when grading, and instructors still need to make great efforts to supervise the process and ensure that students give fair scores to their peers. In addition, the social relationship between students could lead to subjective assessments and affect the fairness of grading. We present the peer review approach to grading projects in computer courses. First, we design a grading specification to improve the accuracy of scoring. Second, we propose a score calculation method based on Technique for Order Preference by Similarity to an Ideal Solution (TOPSIS) to calculate the assignment's final score, which is a multi-objective decision-making problem. Third, we give an anomaly detection algorithm based on Dixon's Q test to filter peer reviewers' unreasonable scores. Based on the approach, we designed a peer review workflow and developed an online peer review tool called Peer Review Studio (PRS). We carry out the method in the undergraduate computer course of 2022. We collect and analyze the learning data between 2020 and 2022. When measured by Krippendorff's alpha, the inter-rater reliability between instructor and peer grading is in good agreement. When measured by Kruskal-Wallis, students' project performance is significantly improved in the first year of peer review. Although learning engagement increases in 2022, there is no statistically significant difference with 2021. The course questionnaire of 2022 shows that most students approve of the peer review approach.