Abstract:The integration of AI assistants, especially through the development of Large Language Models (LLMs), into computer science education has sparked significant debate. An emerging body of work has looked into using LLMs in education, but few have examined the impacts of LLMs on students in entry-level programming courses, particularly in real-world contexts and over extended periods. To address this research gap, we conducted a semester-long, between-subjects study with 50 students using CodeTutor, an LLM-powered assistant developed by our research team. Our study results show that students who used CodeTutor (the experimental group) achieved statistically significant improvements in their final scores compared to peers who did not use the tool (the control group). Within the experimental group, those without prior experience with LLM-powered tools demonstrated significantly greater performance gain than their counterparts. We also found that students expressed positive feedback regarding CodeTutor's capability, though they also had concerns about CodeTutor's limited role in developing critical thinking skills. Over the semester, students' agreement with CodeTutor's suggestions decreased, with a growing preference for support from traditional human teaching assistants. Our analysis further reveals that the quality of user prompts was significantly correlated with CodeTutor's response effectiveness. Building upon our results, we discuss the implications of our findings for integrating Generative AI literacy into curricula to foster critical thinking skills and turn to examining the temporal dynamics of user engagement with LLM-powered tools. We further discuss the discrepancy between the anticipated functions of tools and students' actual capabilities, which sheds light on the need for tailored strategies to improve educational outcomes.

"Real Learner Data Matters" Exploring the Design of LLM-Powered Question Generation for Deaf and Hard of Hearing Learners

LLM-Powered AI Tutors with Personas for d/Deaf and Hard-of-Hearing Online Learners

Spoken Language Intelligence of Large Language Models for Language Learning

Adapting Large Language Models for Education: Foundational Capabilities, Potentials, and Challenges

Inclusive Emotion Technologies: Addressing the Needs of d/Deaf and Hard of Hearing Learners in Video-Based Learning

Large Language Models for Education: A Survey and Outlook

AI-TA: Towards an Intelligent Question-Answer Teaching Assistant using Open-Source LLMs

Evaluating the Effectiveness of LLMs in Introductory Computer Science Education: A Semester-Long Field Study

Let the LLMs Talk: Simulating Human-to-Human Conversational QA via Zero-Shot LLM-to-LLM Interactions

Dr.Academy: A Benchmark for Evaluating Questioning Capability in Education for Large Language Models

Improving Bilingual Capabilities of Language Models to Support Diverse Linguistic Practices in Education

Large Language Models Meet User Interfaces: The Case of Provisioning Feedback

"We do use it, but not how hearing people think": How the Deaf and Hard of Hearing Community Uses Large Language Model Tools

Knowledge-Infused Legal Wisdom: Navigating LLM Consultation through the Lens of Diagnostics and Positive-Unlabeled Reinforcement Learning

Designing Child-Centric AI Learning Environments: Insights from LLM-Enhanced Creative Project-Based Learning

Leveraging Language Models for Emotion and Behavior Analysis in Education

Empowering Language Models with Active Inquiry for Deeper Understanding

Leveraging Large Language Models to Generate Course-specific Semantically Annotated Learning Objects

Supervised Knowledge Makes Large Language Models Better In-context Learners

Insights from Social Shaping Theory: The Appropriation of Large Language Models in an Undergraduate Programming Course