Abstract:Providing timely and personalized feedback to large numbers of students is a long-standing challenge in programming courses. Relying on human teaching assistants (TAs) has been extensively studied, revealing a number of potential shortcomings. These include inequitable access for students with low confidence when needing support, as well as situations where TAs provide direct solutions without helping students to develop their own problem-solving skills. With the advent of powerful large language models (LLMs), digital teaching assistants configured for programming contexts have emerged as an appealing and scalable way to provide instant, equitable, round-the-clock support. Although digital TAs can provide a variety of help for programming tasks, from high-level problem solving advice to direct solution generation, the effectiveness of such tools depends on their ability to promote meaningful learning experiences. If students find the guardrails implemented in digital TAs too constraining, or if other expectations are not met, they may seek assistance in ways that do not help them learn. Thus, it is essential to identify the features that students believe make digital teaching assistants valuable. We deployed an LLM-powered digital assistant in an introductory programming course and collected student feedback ($n=813$) on the characteristics of the tool they perceived to be most important. Our results highlight that students value such tools for their ability to provide instant, engaging support, particularly during peak times such as before assessment deadlines. They also expressed a strong preference for features that enable them to retain autonomy in their learning journey, such as scaffolding that helps to guide them through problem-solving steps rather than simply being shown direct solutions.
What problem does this paper attempt to address?
This paper attempts to address the long-standing challenge of providing timely, personalized feedback in programming education. Specifically, the authors focus on how to leverage large language models (LLMs) to construct digital teaching assistants to replace or supplement the functions of human teaching assistants (TAs). While human TAs play a crucial role in providing support, they have some limitations, such as being unable to meet all students' needs and sometimes giving direct answers instead of guiding students to solve problems. Therefore, the paper investigates students' acceptance and preferences for a digital teaching assistant (CodeHelp) based on LLMs to determine which features students consider most important.
### Main Questions
1. **How do students perceive the effectiveness of the digital teaching assistant's responses in debugging and coding tasks?**
- The first research question is to evaluate students' perceptions of the correctness and helpfulness of the digital teaching assistant's responses in debugging and coding tasks.
2. **Which attributes do students consider most important for supporting their learning in introductory programming courses?**
- The second research question is to understand which features students believe are most necessary for a digital programming assistant to better support their learning.
### Methods
- **Course Background**: The study was conducted in a large introductory programming course at the University of Auckland, New Zealand, with a total of 889 students participating.
- **Tool**: The digital teaching assistant CodeHelp, based on GPT-4, was used. This tool can answer students' free-form queries but does not generate solution code.
- **Experimental Design**: Students observed CodeHelp's responses to debugging and coding problems during lab activities and evaluated the quality of these responses afterward.
- **Data Collection**: Students completed questionnaires after finishing lab tasks, assessing whether CodeHelp's responses were correct and helpful, and describing the important features they believe a digital teaching assistant should have.
### Results
- **Interaction**: During the two-week study period, 813 students submitted 6,051 queries. Most students found CodeHelp's responses to be both correct and helpful.
- **Important Features**:
- **Scaffolding**: Students wanted the assistant to guide them step-by-step in solving problems rather than providing direct answers. They emphasized the importance of understanding the process and foundational knowledge.
- **Appropriateness**: The explanations and feedback should be suitable for the students' level, avoiding complex terminology, and ensuring specificity and accuracy of the information.
- **User Experience**: Students desired the tool to provide immediate support, be interactive, and be trustworthy in its feedback.
### Discussion
- **Cost-Effectiveness**: The digital teaching assistant is relatively low-cost and can provide continuous support 24/7, which is especially useful during non-working hours.
- **Future Research Directions**: Future research could explore the impact of always-available support on students' procrastination behavior and how to further optimize the design of digital teaching assistants to enhance learning outcomes.
In summary, this paper, through empirical research, reveals students' expectations and preferences for digital teaching assistants, providing valuable insights for developing more effective programming education support tools.