Abstract:Abstract Generative AI has prompted educators to reevaluate traditional teaching and assessment methods. This study examines AI’s ability to write essays analysing Old English poetry; human markers assessed and attempted to distinguish them from authentic analyses of poetry by first-year undergraduate students in English at the University of Oxford. Using the standard UK University grading system, AI-written essays averaged a score of 60.46, whilst human essays achieved 63.57, a margin of difference not statistically significant ( p = 0.10). Notably, student submissions applied a nuanced understanding of cultural context and secondary criticism to their close reading, while AI essays often described rather than analysed, lacking depth in the evaluation of poetic features, and sometimes failing to properly recognise key aspects of passages. Distinguishing features of human essays included detailed and sustained analysis of poetic style, as well as spelling errors and lack of structural cohesion. AI essays, on the other hand, exhibited a more formal structure and tone but sometimes fell short in incisive critique of poetic form and effect. Human markers correctly identified the origin of essays 79.41% of the time. Additionally, we compare three purported AI detectors, finding that the best, ‘Quillbot’, correctly identified the origin of essays 95.59% of the time. However, given the high threshold for academic misconduct, conclusively determining origin remains challenging. The research also highlights the potential benefits of generative AI’s ability to advise on structuring essays and suggesting avenues for research. We advocate for transparency regarding AI’s capabilities and limitations, and this study underscores the importance of human critical engagement in teaching and learning in Higher Education. As AI’s proficiency grows, educators must reevaluate what authentic assessment is, and consider implementing dynamic, holistic methods to ensure academic integrity.

Grading the Graders: Comparing Generative AI and Human Assessment in Essay Evaluation

Student and Faculty Perceptions of Generative Artificial Intelligence in Student Writing

AI, write an essay for me: A large-scale comparison of human-written versus ChatGPT-generated essays

ChatGPT versus human essayists: an exploration of the impact of artificial intelligence for authorship and academic integrity in the humanities

Using Generative AI to Promote Psychological, Feedback, and Artificial Intelligence Literacies in Undergraduate Psychology

AI-generated feedback on writing: insights into efficacy and ENL student preference

Beyond human subjectivity and error: a novel AI grading system

Is AI changing learning and assessment as we know it? Evidence from a ChatGPT experiment and a conceptual framework

Testing the Ability of Teachers and Students to Differentiate between Essays Generated by ChatGPT and High School Students

Generative Grading: Near Human-level Accuracy for Automated Feedback on Richly Structured Problems

Large Language Models in Student Assessment: Comparing ChatGPT and Human Graders

Evaluating AI-Generated Essays with GRE Analytical Writing Assessment

Comparative Analysis of GPT-4 and Human Graders in Evaluating Praise Given to Students in Synthetic Dialogues

Not quite eye to A.I.: student and teacher perspectives on the use of generative artificial intelligence in the writing process

Digital Ink and Surgical Dreams: Perceptions of Artificial Intelligence-Generated Essays in Residency Applications

Generative AI Usage and Exam Performance

ChatGPT-3.5 as writing assistance in students' essays

Assessing Confidence in AI-Assisted Grading of Physics Exams through Psychometrics: An Exploratory Study

Improving Writing Feedback for Struggling Writers: Generative AI to the Rescue?

Transforming Educational Assessment: Insights Into the Use of ChatGPT and Large Language Models in Grading

Effects of Human vs. Automatic Feedback on Students' Understanding of AI Concepts and Programming Style