Abstract:Text-based open-ended questions in academic formative and summative assessments help students become deep learners and prepare them to understand concepts for a subsequent conceptual assessment. However, grading text-based questions, especially in large (>50 enrolled students) courses, is tedious and time-consuming for instructors. Text processing models continue progressing with the rapid development of Artificial Intelligence (AI) tools and Natural Language Processing (NLP) algorithms. Especially after breakthroughs in Large Language Models (LLM), there is immense potential to automate rapid assessment and feedback of text-based responses in education. This systematic review adopts a scientific and reproducible literature search strategy based on the PRISMA process using explicit inclusion and exclusion criteria to study text-based automatic assessment systems in post-secondary education, screening 838 papers and synthesizing 93 studies. To understand how text-based automatic assessment systems have been developed and applied in education in recent years, three research questions are considered: 1) What types of automated assessment systems can be identified using input, output, and processing framework? 2) What are the educational focus and research motivations of studies with automated assessment systems? 3) What are the reported research outcomes in automated assessment systems and the next steps for educational applications? All included studies are summarized and categorized according to a proposed comprehensive framework, including the input and output of the system, research motivation, and research outcomes, aiming to answer the research questions accordingly. Additionally, the typical studies of automated assessment systems, research methods, and application domains in these studies are investigated and summarized. This systematic review provides an overview of recent educational applications of text-based assessment systems for understanding the latest AI/NLP developments assisting in text-based assessments in higher education. Findings will particularly benefit researchers and educators incorporating LLMs such as ChatGPT into their educational activities.

Machine Learning Mixed Methods Text Analysis: An Illustration From Automated Scoring Models of Student Writing in Biology Education

Automatic short answer grading and feedback using text mining methods

Assessing Students’ Use of Evidence and Organization in Response-to-Text Writing: Using Natural Language Processing for Rubric-Based Automated Scoring

Applying machine learning in science assessment: a systematic review

Design and Implementation of Machine Learning Algorithms in Automatic Grading of Students' Assignments

Using Text Mining and Data Mining Techniques for Applied Learning Assessment

Automatic assessment of text-based responses in post-secondary education: A systematic review

Automated Assessment of Multimodal Answer Sheets in the STEM domain

Text as Data Methods for Education Research

Machine Learning-Based Intelligent Scoring System for English Essays under the Background of Modern Information Technology

Using machine learning for continuous updating of meta-analysis in educational context

A Study of Automated Evaluation of Student’s Examination Paper using Machine Learning Techniques

Construction of Intelligent Evaluation Model of English Composition Based on Machine Learning

Simulation of E-learning in English learner writing teaching based on natural language processing and machine learning algorithms

Improving predictive power through deep learning analysis of K-12 online student behaviors and discussion board content

A Multi-Strategy Computer-Assisted EFL Writing Learning System With Deep Learning Incorporated and Its Effects on Learning: A Writing Feedback Perspective

Machine learning to detect invalid text responses: Validation and comparison to existing detection methods

An Examination of the Use of Large Language Models to Aid Analysis of Textual Data

Automatic Scoring of Students' Science Writing Using Hybrid Neural Network

College English Intelligent Writing Score System Based on Big Data Analysis and Deep Learning Algorithm