Abstract:This article investigates the feasibility of using automated scoring methods to evaluate the quality of student-written essays. In 2012, Kaggle hosted an Automated Student Assessment Prize contest to find effective solutions to automated testing and grading. This article: a) analyzes the datasets from the contest – which contained hand-graded essays – to measure their suitability for developing competent automated grading tools; b) evaluates the potential for deep learning in automated essay scoring (AES) to produce sophisticated testing and grading algorithms; c) advocates for thorough and transparent performance reports on AES research, which will facilitate fairer comparisons among various AES systems and permit study replication; d) uses both deep neural networks and state-of-the-art NLP tools to predict finer-grained rubric scores, to illustrate how rubric scores are determined from a linguistic perspective, and to uncover important features of an effective rubric scoring model. This study's findings first highlight the level of agreement that exists between two human raters for each rubric as captured in the investigated essay dataset, that is, 0.60 on average as measured by the quadratic weighted kappa (QWK). Only one related study has been found in the literature which also performed rubric score predictions through models trained on the same dataset. At best, the predictive models had an average agreement level (QWK) of 0.53 with the human raters, below the level of agreement among human raters. In contrast, this research's findings report an average agreement level per rubric with the two human raters' resolved scores of 0.72 (QWK), well beyond the agreement level between the two human raters. Further, the AES system proposed in this article predicts holistic essay scores through its predicted rubric scores and produces a QWK of 0.78, a competitive performance according to recent literature where cutting-edge AES tools generate agreement levels between 0.77 and 0.81, results computed as per the same procedure as in this article. This study's AES system goes one step further toward interpretability and the provision of high-level explanations to justify the predicted holistic and rubric scores. It contends that predicting rubric scores is essential to automated essay scoring, because it reveals the reasoning behind AIED-based AES systems. Will building AIED accountability improve the trustworthiness of the formative feedback generated by AES? Will AIED-empowered AES systems thoroughly mimic, or even outperform, a competent human rater? Will such machine-grading systems be subjected to verification by human raters, thus paving the way for a human-in-the-loop assessment mechanism? Will trust in new generations of AES systems be improved with the addition of models that explain the inner workings of a deep learning black box? This study seeks to expand these horizons of AES to make the technique practical, explainable, and trustable.

Using Writing Process and Product Features to Assess Writing Quality and Explore How Those Features Relate to Other Literacy Tasks. Research Report. ETS RR-14-03.

Evaluating Writing Process Features in an Adult EFL Writing Assessment Context: A Keystroke Logging Study

Monitoring of Scoring Using the E‐rater® Automated Scoring System and Human Raters on a Writing Test

What accounts for integrated reading-to-write task scores?

Application of an Automated Essay Scoring engine to English writing assessment using Many-Facet Rasch Measurement

Automating individualized, process-focused writing instruction: A design-based research study

The role of automated writing evaluation holistic scores in the ESL classroom

Examining the Effects of a Real-Time, Knowledge-Aware Tool for Academic Writing Assessment

Natural language processing in an intelligent writing strategy tutoring system

Assessing Students’ Use of Evidence and Organization in Response-to-Text Writing: Using Natural Language Processing for Rubric-Based Automated Scoring

Automatic Essay Assessment

A Comparison of EFL Raters' Essay-Rating Processes across Two Types of Rating Scales.

Rater cognitive processes in integrated writing tasks: from the perspective of problem-solving

Using the Developmental Path of Cause to Bridge the Gap Between Awe Scores and Writing Teachers' Evaluations

The Sensitivity of a Scenario-Based Assessment of Written Argumentation to School Differences in Curriculum and Instruction

Toward a Transparent Construct of Reading-to-Write Tasks: The Interface Between Discourse Features and Proficiency

Improving EFL Graduate Students ’ Proficiency in Writing Through an Online Automated Essay Assessing System

The affectability of writing assessment scores: a G-theory analysis of rater, task, and scoring method contribution

Automated language essay scoring systems: a literature review

Automated Essay Scoring and the Deep Learning Black Box: How Are Rubric Scores Determined?

Automated assessment of non-native learner essays: Investigating the role of linguistic features