Abstract:The integration of AI assistants, especially through the development of Large Language Models (LLMs), into computer science education has sparked significant debate. An emerging body of work has looked into using LLMs in education, but few have examined the impacts of LLMs on students in entry-level programming courses, particularly in real-world contexts and over extended periods. To address this research gap, we conducted a semester-long, between-subjects study with 50 students using CodeTutor, an LLM-powered assistant developed by our research team. Our study results show that students who used CodeTutor (the experimental group) achieved statistically significant improvements in their final scores compared to peers who did not use the tool (the control group). Within the experimental group, those without prior experience with LLM-powered tools demonstrated significantly greater performance gain than their counterparts. We also found that students expressed positive feedback regarding CodeTutor's capability, though they also had concerns about CodeTutor's limited role in developing critical thinking skills. Over the semester, students' agreement with CodeTutor's suggestions decreased, with a growing preference for support from traditional human teaching assistants. Our analysis further reveals that the quality of user prompts was significantly correlated with CodeTutor's response effectiveness. Building upon our results, we discuss the implications of our findings for integrating Generative AI literacy into curricula to foster critical thinking skills and turn to examining the temporal dynamics of user engagement with LLM-powered tools. We further discuss the discrepancy between the anticipated functions of tools and students' actual capabilities, which sheds light on the need for tailored strategies to improve educational outcomes.

An Empirical Study on Integrated LLS Instruction & SL Scores

The English Learning Motivation of Chinese Secondary School Students: Cross-grade Survey Analysis

The English Learning Motivation of Chinese High School Students: Cross-grade Survey Analysis

Corpus-based Investigation into Communication Strategies in CET- SET

A Study on the Relationship between Language Learning Strategies and Performance in CET-4 in an Internet and Multimedia Environment

Modeling and verification of the structure of College English learning motivation within educational context

The Influence of EFL Proficiency on Written Products in Independent and Integrated Writing Tasks

Linguistic Features Distinguishing Students' Writing Ability Aligned with CEFR Levels

Measuring English Learning Strategy in High School Students with a New Survey and Validating the Instrument with Rasch Theory

A Survey of Learning Strategies of English Majors (Grade 03)

English Language Requirement and Educational Inequality: Evidence from 16 Million College Applicants in China

Killing Two Birds with One Stone? A Study on Achievement Levels and Affective Factors in Content and Language Integrated Learning (CLIL)

The relationship between written discourse features and integrated listening-to-write scores for adolescent English language learners

Exploring the relative contributions of learning motivations and test perceptions to autonomous English as a foreign language learning and achievement

An Empirical Study on English Autonomous Learning Ability of Ethnic Minority Students of Preparatory Education

What Contributes to Student Language Learning Satisfaction and Achievement with Learning Management Systems?

College students’ writing self-efficacy and writing self-regulated learning strategies in learning English as a foreign language

Language learning strategy use in context: the effects of self-efficacy and CLIL on language proficiency

Evaluating the Effectiveness of LLMs in Introductory Computer Science Education: A Semester-Long Field Study

A study of the Relationship Between High School Students' English Learning Self-Efficacy, Anxiety and Learning Effectiveness

Reading motivation and learning strategies among Chinese EFL learners