Abstract:This research to practice full paper presented a peer review approach to grading projects in computer courses. Educators commonly adopt project-based learning activities in computer courses to achieve the goal. Implementations of these projects typically involve hardly quantifiable dimensions, such as novelty, functionality, user-friendly, coding style, and document. Automated tools are not competent for project assessment. However, assessing projects by instructors can be labor intensive and generally involves a high degree of subjectivity. Peer review is widely used in high education as effective active learning in computer courses, especially in the past decade. Despite its many advantages, peer review still faces some challenges. Students need to be more expert in assessment and might make mistakes when grading, and instructors still need to make great efforts to supervise the process and ensure that students give fair scores to their peers. In addition, the social relationship between students could lead to subjective assessments and affect the fairness of grading. We present the peer review approach to grading projects in computer courses. First, we design a grading specification to improve the accuracy of scoring. Second, we propose a score calculation method based on Technique for Order Preference by Similarity to an Ideal Solution (TOPSIS) to calculate the assignment's final score, which is a multi-objective decision-making problem. Third, we give an anomaly detection algorithm based on Dixon's Q test to filter peer reviewers' unreasonable scores. Based on the approach, we designed a peer review workflow and developed an online peer review tool called Peer Review Studio (PRS). We carry out the method in the undergraduate computer course of 2022. We collect and analyze the learning data between 2020 and 2022. When measured by Krippendorff's alpha, the inter-rater reliability between instructor and peer grading is in good agreement. When measured by Kruskal-Wallis, students' project performance is significantly improved in the first year of peer review. Although learning engagement increases in 2022, there is no statistically significant difference with 2021. The course questionnaire of 2022 shows that most students approve of the peer review approach.

Automated Evaluation for Performance Test Scripts

PowerGrader: Automating Code Assessment Based on PowerShell for Programming Courses.

Automated Student Code Scoring by Analyzing Grammatical and Semantic Information of Code

Research on Automatic Scoring for Java Programming Questions

META: Multidimensional Evaluation of Testing Ability

A Peer Review Approach to Grading Projects in Computer Courses.

An Analysis of Programming Course Evaluations Before and After the Introduction of an Autograder

Examining the Effects of a Real-Time, Knowledge-Aware Tool for Academic Writing Assessment

An automated essay scoring systems: a systematic literature review

Automated Assessment in Computer Science Education: A State-of-the-Art Review

Automated Grading and Feedback Tools for Programming Education: A Systematic Review

Adaptive online training and evaluation systems

Automated Computer Program Evaluation and Projects -- Our Experiences

Automatic Grading of Student Code with Similarity Measurement

A Method of Computer Automatic Scoring for Subjective Questions

Automated Writing Evaluation Using Sentence by Sentence Scoring Model

A Peer Grading Approach for Open-ended Programming Projects Based on Binary System and Swiss System

Validity of a graph-based automatic assessment system for programming assignments: human versus automatic grading

Neural Automated Writing Evaluation with Corrective Feedback

Intelligent On-line Exam Management and Evaluation System