AI Meets the Classroom: When Does ChatGPT Harm Learning?

Matthias Lehmann,Philipp B. Cornelius,Fabian J. Sting

2024-08-30

Abstract:In this paper, we study how generative AI and specifically large language models (LLMs) impact learning in coding classes. We show across three studies that LLM usage can have positive and negative effects on learning outcomes. Using observational data from university-level programming courses, we establish such effects in the field. We replicate these findings in subsequent experimental studies, which closely resemble typical learning scenarios, to show causality. We find evidence for two contrasting mechanisms that determine the overall effect of LLM usage on learning. Students who use LLMs as personal tutors by conversing about the topic and asking for explanations benefit from usage. However, learning is impaired for students who excessively rely on LLMs to solve practice exercises for them and thus do not invest sufficient own mental effort. Those who never used LLMs before are particularly prone to such adverse behavior. Students without prior domain knowledge gain more from having access to LLMs. Finally, we show that the self-perceived benefits of using LLMs for learning exceed the actual benefits, potentially resulting in an overestimation of one's own abilities. Overall, our findings show promising potential of LLMs as learning support, however also that students have to be very cautious of possible pitfalls.

Computers and Society,Artificial Intelligence,Human-Computer Interaction,Machine Learning

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to explore how generative artificial intelligence (AI), especially large language models (LLMs), affect students' learning effectiveness in programming courses. Specifically, researchers conducted three studies (including field data and laboratory experiments) to explore the positive and negative impacts of LLM use on students' learning outcomes. The study found that students can benefit from the use of LLMs when they use LLMs as personal tutors, understand topics through conversations and request explanations; however, if students rely too much on LLMs to solve exercise problems, it will harm their learning effectiveness. In addition, students who have never used LLMs are more likely to exhibit this negative behavior, and students without prior domain knowledge benefit more from accessing LLMs. Finally, the study also pointed out that students' self - perceived benefits of using LLMs for learning exceed the actual benefits, which may lead to overestimation of their own abilities. Overall, although LLMs have great potential as learning support tools, students must be very cautious to avoid possible pitfalls.