Exploring How Multiple Levels of GPT-Generated Programming Hints Support or Disappoint Novices

Ruiwei Xiao,Xinying Hou,John Stamper

DOI: https://doi.org/10.1145/3613905.3650937

2024-04-03

Abstract:Recent studies have integrated large language models (LLMs) into diverse educational contexts, including providing adaptive programming hints, a type of feedback focuses on helping students move forward during problem-solving. However, most existing LLM-based hint systems are limited to one single hint type. To investigate whether and how different levels of hints can support students' problem-solving and learning, we conducted a think-aloud study with 12 novices using the LLM Hint Factory, a system providing four levels of hints from general natural language guidance to concrete code assistance, varying in format and granularity. We discovered that high-level natural language hints alone can be helpless or even misleading, especially when addressing next-step or syntax-related help requests. Adding lower-level hints, like code examples with in-line comments, can better support students. The findings open up future work on customizing help responses from content, format, and granularity levels to accurately identify and meet students' learning needs.

Human-Computer Interaction,Artificial Intelligence,Computers and Society

What problem does this paper attempt to address?

The paper attempts to address the issue of how to effectively support beginners in problem-solving and learning during programming education through different levels of GPT-generated programming hints. Existing hint systems based on large language models (LLMs) are mostly limited to a single type of hint, whereas this paper aims to explore how different levels of hints can better support students' problem-solving and learning processes.

Specifically, the paper developed a system called "LLM Hint Factory," which can provide four different levels of hints ranging from general natural language guidance to specific code snippet suggestions. Through an experiment involving 12 programming beginners, the researchers found that providing only high-level natural language hints is sometimes insufficient and may even mislead students, especially when dealing with next-step logic or syntax-related issues. In contrast, adding lower-level hints (such as annotated code examples) was more helpful in most cases for students to solve problems. This finding offers new insights for the design of future teaching agents, emphasizing the importance of personalized responses based on the specific help-seeking context of students.