Abstract:Automatic Essay Scoring (AES) is a well-established educational pursuit that employs machine learning to evaluate student-authored essays. While much effort has been made in this area, current research primarily focuses on either (i) boosting the predictive accuracy of an AES model for a specific prompt (i.e., developing prompt-specific models), which often heavily relies on the use of the labeled data from the same target prompt; or (ii) assessing the applicability of AES models developed on non-target prompts to the intended target prompt (i.e., developing the AES models in a cross-prompt setting). Given the inherent bias in machine learning and its potential impact on marginalized groups, it is imperative to investigate whether such bias exists in current AES methods and, if identified, how it intervenes with an AES model's accuracy and generalizability. Thus, our study aimed to uncover the intricate relationship between an AES model's accuracy, fairness, and generalizability, contributing practical insights for developing effective AES models in real-world education. To this end, we meticulously selected nine prominent AES methods and evaluated their performance using seven metrics on an open-sourced dataset, which contains over 25,000 essays and various demographic information about students such as gender, English language learner status, and economic status. Through extensive evaluations, we demonstrated that: (1) prompt-specific models tend to outperform their cross-prompt counterparts in terms of predictive accuracy; (2) prompt-specific models frequently exhibit a greater bias towards students of different economic statuses compared to cross-prompt models; (3) in the pursuit of generalizability, traditional machine learning models coupled with carefully engineered features hold greater potential for achieving both high accuracy and fairness than complex neural network models.

Multi-Stage Pre-training for Automated Chinese Essay Scoring

An Automated Grader for Chinese Essay Combining Shallow and Deep Semantic Attributes

Attention-based Recurrent Convolutional Neural Network for Automatic Essay Scoring.

Automated Cross-prompt Scoring of Essay Traits

Automated essay scoring based on the enhanced chimp optimization algorithm-back propagation (ENChOA-BP) and K-means

Research on the optimization of essay scores based on deep learning

Automated Essay Scoring based on Two-Stage Learning

Prompt Agnostic Essay Scorer: A Domain Generalization Approach to Cross-prompt Automated Essay Scoring

Prompt- and Trait Relation-aware Cross-prompt Essay Trait Scoring

Computer-Aided English Education in China: An Online Automatic Essay Scoring System

On the Use of BERT for Automated Essay Scoring: Joint Learning of Multi-Scale Essay Representation

Automatic Essay Multi-dimensional Scoring with Fine-tuning and Multiple Regression

An intelligent scoring system for English writing based on artificial intelligence and machine learning

Automated Essay Scoring Using Grammatical Variety and Errors with Multi-Task Learning and Item Response Theory

Applying Large Language Models for Automated Essay Scoring for Non-Native Japanese

Machine Learning-Based Intelligent Scoring System for English Essays under the Background of Modern Information Technology

Task-Related Pretraining with Whole Word Masking for Chinese Coherence Evaluation.

Improving Performance of Automated Essay Scoring by using back-translation essays and adjusted scores

Unveiling the Tapestry of Automated Essay Scoring: A Comprehensive Investigation of Accuracy, Fairness, and Generalizability

reStructured Pre-training

Empirical Study of Large Language Models as Automated Essay Scoring Tools in English Composition__Taking TOEFL Independent Writing Task for Example