The Influence of GenAI on the Effectiveness of Argumentative Writing in Higher Education: Evidence from a Quasi-Experimental Study in China

Hanlin Li,Yu Wang,Siqi Luo,Cui Huang

DOI: https://doi.org/10.1080/17516234.2024.2363128

2024-01-01

Journal of Asian Public Policy

Abstract:The revolutionary development of generative artificial intelligence, especially large language model (LLM) chatbots like ChatGPT, has brought disruptive changes to society, especially in the field of higher education, particularly in the domain of writing. Prior studies have primarily examined the opportunities and challenges that LLM Chatbots bring to higher education writing from a theoretical perspective. However, empirical studies on how to effectively and responsibly integrate LLM Chatbots into writing learning and evaluate their effectiveness are still limited. To address this gap, the present study introduces a learning approach termed LLM-powered Chatbot-assisted argumentative writing (LCAW) to support the cultivation of university students' writing abilities. A total of 61 Chinese university students from two classes were invited to participate in a quasi-experimental study with different learning methods as intervention measures, and student perceptions were gathered through interviews. The results show that the proposed LCAW method significantly improves students' writing performance in terms of content quality and language expression, and positively impacts in providing personalized feedback and enhancing writing motivation. This study contributes by proposing an effective method for integrating generative AI into higher education writing instruction and by offering insights for curriculum design policies to enhance higher writing education through GenAI.