Abstract:Open-ended questions, which require students to produce multi-word, nontrivial responses, are a popular tool for formative assessment as they provide more specific insights into what students do and don't know. However, grading open-ended questions can be time-consuming leading teachers to resort to simpler question formats or conduct fewer formative assessments. While there has been a longstanding interest in automating of short-answer grading (ASAG), but previous approaches have been technically complex, limiting their use in formative assessment contexts. The newest generation of Large Language Models (LLMs) potentially makes grading short answer questions more feasible. This paper investigates the potential for the newest version of LLMs to be used in ASAG, specifically in the grading of short answer questions for formative assessments, in two ways. First, it introduces a novel dataset of short answer reading comprehension questions, drawn from a set of reading assessments conducted with over 150 students in Ghana. This dataset allows for the evaluation of LLMs in a new context, as they are predominantly designed and trained on data from high-income North American countries. Second, the paper empirically evaluates how well various configurations of generative LLMs grade student short answer responses compared to expert human raters. The findings show that GPT-4, with minimal prompt engineering, performed extremely well on grading the novel dataset (QWK 0.92, F1 0.89), reaching near parity with expert human raters. To our knowledge this work is the first to empirically evaluate the performance of generative LLMs on short answer reading comprehension questions using real student data, with low technical hurdles to attaining this performance. These findings suggest that generative LLMs could be used to grade formative literacy assessment tasks.

Post-Hoc Answer Attribution for Grounded and Trustworthy Long Document Comprehension: Task, Insights, and Challenges

Enhancing Post-Hoc Attributions in Long Document Comprehension via Coarse Grained Answer Decomposition

Attribute or Abstain: Large Language Models as Long Document Assistants

A Claim Decomposition Benchmark for Long-form Answer Verification

Attributed Question Answering: Evaluation and Modeling for Attributed Large Language Models

Peering into the Mind of Language Models: An Approach for Attribution in Contextual Question Answering

Enhancing Answer Attribution for Faithful Text Generation with Large Language Models

Atomic Fact Decomposition Helps Attributed Question Answering

Detect, Retrieve, Comprehend: A Flexible Framework for Zero-Shot Document-Level Question Answering

Understanding Retrieval Augmentation for Long-Form Question Answering

Investigating Answerability of LLMs for Long-Form Question Answering

Automatic Generation of Short Answer Questions for Reading Comprehension Assessment

Benchmarking Large Language Models in Complex Question Answering Attribution using Knowledge Graphs

Contri(e)ve: Context + Retrieve for Scholarly Question Answering

End-to-End Multihop Retrieval for Compositional Question Answering over Long Documents

Can LLMs Grade Short-Answer Reading Comprehension Questions : An Empirical Study with a Novel Dataset

Adaptive Question Answering: Enhancing Language Model Proficiency for Addressing Knowledge Conflicts with Source Citations

Long-Span Question-Answering: Automatic Question Generation and QA-System Ranking via Side-by-Side Evaluation

How Many Answers Should I Give? an Empirical Study of Multi-Answer Reading Comprehension.

TruthReader: Towards Trustworthy Document Assistant Chatbot with Reliable Attribution