Evaluating the Effectiveness of LLMs in Introductory Computer Science Education: A Semester-Long Field Study

Wenhan Lyu,Yimeng Wang,Tingting,Chung,Yifan Sun,Yixuan Zhang

DOI: https://doi.org/10.1145/3657604.3662036

2024-05-03

Abstract:The integration of AI assistants, especially through the development of Large Language Models (LLMs), into computer science education has sparked significant debate. An emerging body of work has looked into using LLMs in education, but few have examined the impacts of LLMs on students in entry-level programming courses, particularly in real-world contexts and over extended periods. To address this research gap, we conducted a semester-long, between-subjects study with 50 students using CodeTutor, an LLM-powered assistant developed by our research team. Our study results show that students who used CodeTutor (the experimental group) achieved statistically significant improvements in their final scores compared to peers who did not use the tool (the control group). Within the experimental group, those without prior experience with LLM-powered tools demonstrated significantly greater performance gain than their counterparts. We also found that students expressed positive feedback regarding CodeTutor's capability, though they also had concerns about CodeTutor's limited role in developing critical thinking skills. Over the semester, students' agreement with CodeTutor's suggestions decreased, with a growing preference for support from traditional human teaching assistants. Our analysis further reveals that the quality of user prompts was significantly correlated with CodeTutor's response effectiveness. Building upon our results, we discuss the implications of our findings for integrating Generative AI literacy into curricula to foster critical thinking skills and turn to examining the temporal dynamics of user engagement with LLM-powered tools. We further discuss the discrepancy between the anticipated functions of tools and students' actual capabilities, which sheds light on the need for tailored strategies to improve educational outcomes.

Human-Computer Interaction

What problem does this paper attempt to address?

This paper discusses the utility of large language models (LLMs) in introductory computer science education through a semester-long field study. The study found that students using the CodeTutor tool driven by LLMs showed significant improvements in their final grades compared to students who did not use the tool, especially for those with no prior experience with LLM tools. Students provided positive feedback on CodeTutor's ability to understand queries and help learn programming syntax, but expressed concerns about its ability to foster critical thinking skills. As the semester progressed, students became less reliant on CodeTutor and leaned more towards traditional TA support. The study also revealed a significant correlation between the quality of user prompts and the effectiveness of CodeTutor's responses, emphasizing the need to incorporate generative AI literacy into the curriculum to cultivate critical thinking and address the dynamics of long-term student interaction with LLM tools. The paper concludes by discussing the gap between the expected functionalities of the tool and the actual abilities of the students, as well as how adaptive strategies can be developed to improve educational outcomes.

Evaluating the Effectiveness of LLMs in Introductory Computer Science Education: A Semester-Long Field Study

Evaluating the Impact of Advanced LLM Techniques on AI-Lecture Tutors for a Robotics Course

Insights from Social Shaping Theory: The Appropriation of Large Language Models in an Undergraduate Programming Course

When Young Scholars Cooperate with LLMs in Academic Tasks: The Influence of Individual Differences and Task Complexities

CS1-LLM: Integrating LLMs into CS1 Instruction

LLMs are Imperfect, Then What? An Empirical Study on LLM Failures in Software Engineering

"Which LLM should I use?": Evaluating LLMs for tasks performed by Undergraduate Computer Science Students

Navigating the Pitfalls: Analyzing the Behavior of LLMs as a Coding Assistant for Computer Science Students—A Systematic Review of the Literature

Leveraging LLM Tutoring Systems for Non-Native English Speakers in Introductory CS Courses

Impact of Guidance and Interaction Strategies for LLM Use on Learner Performance and Perception

Enhancing LLM-Based Feedback: Insights from Intelligent Tutoring Systems and the Learning Sciences

How Beginning Programmers and Code LLMs (Mis)read Each Other

CourseAssist: Pedagogically Appropriate AI Tutor for Computer Science Education

CodeAid: Evaluating a Classroom Deployment of an LLM-based Programming Assistant that Balances Student and Educator Needs

The Practice of Enhancing Learning and Scientific Innovative Abilities Using LLM-Based AI Tools

LLMs-as-Instructors: Learning from Errors Toward Automating Model Improvement

How to Teach Programming in the AI Era? Using LLMs as a Teachable Agent for Debugging

The GPT Surprise: Offering Large Language Model Chat in a Massive Coding Class Reduced Engagement but Increased Adopters Exam Performances

An Exploratory Study on Upper-Level Computing Students' Use of Large Language Models as Tools in a Semester-Long Project

Study of Artificial Intelligence-Assisted English Oral Teaching

Using an LLM to Help With Code Understanding