Abstract:The integration of AI assistants, especially through the development of Large Language Models (LLMs), into computer science education has sparked significant debate. An emerging body of work has looked into using LLMs in education, but few have examined the impacts of LLMs on students in entry-level programming courses, particularly in real-world contexts and over extended periods. To address this research gap, we conducted a semester-long, between-subjects study with 50 students using CodeTutor, an LLM-powered assistant developed by our research team. Our study results show that students who used CodeTutor (the experimental group) achieved statistically significant improvements in their final scores compared to peers who did not use the tool (the control group). Within the experimental group, those without prior experience with LLM-powered tools demonstrated significantly greater performance gain than their counterparts. We also found that students expressed positive feedback regarding CodeTutor's capability, though they also had concerns about CodeTutor's limited role in developing critical thinking skills. Over the semester, students' agreement with CodeTutor's suggestions decreased, with a growing preference for support from traditional human teaching assistants. Our analysis further reveals that the quality of user prompts was significantly correlated with CodeTutor's response effectiveness. Building upon our results, we discuss the implications of our findings for integrating Generative AI literacy into curricula to foster critical thinking skills and turn to examining the temporal dynamics of user engagement with LLM-powered tools. We further discuss the discrepancy between the anticipated functions of tools and students' actual capabilities, which sheds light on the need for tailored strategies to improve educational outcomes.

What You Need is What You Get: Theory of Mind for an LLM-Based Code Understanding Assistant

Using an LLM to Help With Code Understanding

The Programmer's Assistant: Conversational Interaction with a Large Language Model for Software Development

Evaluating the Effectiveness of LLMs in Introductory Computer Science Education: A Semester-Long Field Study

Intention and Context Elicitation with Large Language Models in the Legal Aid Intake Process

Understanding the Human-LLM Dynamic: A Literature Survey of LLM Use in Programming Tasks

LLM With Tools: A Survey

How Beginning Programmers and Code LLMs (Mis)read Each Other

Why and When LLM-Based Assistants Can Go Wrong: Investigating the Effectiveness of Prompt-Based Interactions for Software Help-Seeking

If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agents

Requirements are All You Need: From Requirements to Code with LLMs

Violation of Expectation via Metacognitive Prompting Reduces Theory of Mind Prediction Error in Large Language Models

LLM-Based Test-Driven Interactive Code Generation: User Study and Empirical Evaluation

Tool-Augmented LLMs as a Universal Interface for IDEs

Learning to Ask: When LLMs Meet Unclear Instruction

Conversational Assistants in Knowledge-Intensive Contexts: An Evaluation of LLM- versus Intent-based Systems

Low-code LLM: Graphical User Interface over Large Language Models

LLMs in the Imaginarium: Tool Learning through Simulated Trial and Error

Mind Your Theory: Theory of Mind Goes Deeper Than Reasoning