Beyond human subjectivity and error: a novel AI grading system

Alexandra Gobrecht,Felix Tuma,Moritz Möller,Thomas Zöller,Mark Zakhvatkin,Alexandra Wuttig,Holger Sommerfeldt,Sven Schütt
2024-05-07
Abstract:The grading of open-ended questions is a high-effort, high-impact task in education. Automating this task promises a significant reduction in workload for education professionals, as well as more consistent grading outcomes for students, by circumventing human subjectivity and error. While recent breakthroughs in AI technology might facilitate such automation, this has not been demonstrated at scale. It this paper, we introduce a novel automatic short answer grading (ASAG) system. The system is based on a fine-tuned open-source transformer model which we trained on large set of exam data from university courses across a large range of disciplines. We evaluated the trained model's performance against held-out test data in a first experiment and found high accuracy levels across a broad spectrum of unseen questions, even in unseen courses. We further compared the performance of our model with that of certified human domain experts in a second experiment: we first assembled another test dataset from real historical exams - the historic grades contained in that data were awarded to students in a regulated, legally binding examination process; we therefore considered them as ground truth for our experiment. We then asked certified human domain experts and our model to grade the historic student answers again without disclosing the historic grades. Finally, we compared the hence obtained grades with the historic grades (our ground truth). We found that for the courses examined, the model deviated less from the official historic grades than the human re-graders - the model's median absolute error was 44 % smaller than the human re-graders', implying that the model is more consistent than humans in grading. These results suggest that leveraging AI enhanced grading can reduce human subjectivity, improve consistency and thus ultimately increase fairness.
Artificial Intelligence
What problem does this paper attempt to address?
The problem that this paper attempts to solve is the subjectivity and error in open - ended question scoring. Specifically, scoring open - ended questions is a highly labor - intensive and significant - impact task in education. Automating this task is expected to significantly reduce the workload of education professionals and, by avoiding human subjectivity and error, provide more consistent scoring results for students. Although recent breakthroughs in artificial intelligence technology may contribute to this automation, it has not been verified in large - scale applications. Therefore, this study aims to develop a new Automatic Short - Answer Grading (ASAG) system to reduce subjectivity and error in the grading process and improve the consistency and fairness of grading. ### Main Contributions 1. **Proposed a new Automatic Short - Answer Grading system**: This system is based on a fine - tuned open - source Transformer model, and the training data comes from university course examination data in multiple disciplines. 2. **Evaluated the performance of the system**: The performance of the system in unseen questions and courses was verified through experiments, and it was found that the system has high accuracy on a wide range of unseen questions. 3. **Compared with human scoring experts**: By re - scoring historical examination data, the scoring consistency between the system and certified human domain experts was compared, and it was found that the system is more consistent than human scorers in most cases. ### Methods - **System Architecture**: The inputs include the question, the reference answer, the maximum score, and the student's answer, and the output is the student's score. - **Fine - tuning method**: The model was fine - tuned using a university examination data set containing multiple disciplines. The data set includes questions, student answers, high - quality reference answers, the maximum score, and official scores. - **Experimental Design**: The performance of the system was evaluated through two experiments: - **Experiment 1**: The performance and generalization ability of the model were evaluated using the reserved test set. - **Experiment 2**: Compared with human scoring experts to evaluate the performance of the model in practical applications. ### Results - **Experiment 1**: The model performed well on unseen questions and courses, with low Mean Absolute Error (MAE) and Root Mean Square Error (RMSE), and a high correlation coefficient. - **Experiment 2**: Compared with human scoring experts, the model has a smaller scoring bias, especially performing better in most courses. The median absolute error of the model is 44% lower than that of human scorers. ### Significance These results indicate that an artificial - intelligence - enhanced scoring system can reduce human subjectivity, improve the consistency of scoring, and ultimately increase the fairness of scoring. This provides strong support for the promotion of future autonomous scoring systems in practical applications.