One Step at a Time: Combining LLMs and Static Analysis to Generate Next-Step Hints for Programming Tasks

Anastasiia Birillo,Elizaveta Artser,Anna Potriasaeva,Ilya Vlasov,Katsiaryna Dzialets,Yaroslav Golubev,Igor Gerasimov,Hieke Keuning,Timofey Bryksin

DOI: https://doi.org/10.1145/3699538.3699556

2024-10-12

Abstract:Students often struggle with solving programming problems when learning to code, especially when they have to do it online, with one of the most common disadvantages of working online being the lack of personalized help. This help can be provided as next-step hint generation, i.e., showing a student what specific small step they need to do next to get to the correct solution. There are many ways to generate such hints, with large language models (LLMs) being among the most actively studied right now. While LLMs constitute a promising technology for providing personalized help, combining them with other techniques, such as static analysis, can significantly improve the output quality. In this work, we utilize this idea and propose a novel system to provide both textual and code hints for programming tasks. The pipeline of the proposed approach uses a chain-of-thought prompting technique and consists of three distinct steps: (1) generating subgoals - a list of actions to proceed with the task from the current student's solution, (2) generating the code to achieve the next subgoal, and (3) generating the text to describe this needed action. During the second step, we apply static analysis to the generated code to control its size and quality. The tool is implemented as a modification to the open-source JetBrains Academy plugin, supporting students in their in-IDE courses. To evaluate our approach, we propose a list of criteria for all steps in our pipeline and conduct two rounds of expert validation. Finally, we evaluate the next-step hints in a classroom with 14 students from two universities. Our results show that both forms of the hints - textual and code - were helpful for the students, and the proposed system helped them to proceed with the coding tasks.

Software Engineering,Artificial Intelligence,Computers and Society,Human-Computer Interaction

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: during the programming learning process, students often encounter difficulties when studying online, especially in the absence of personalized help. Specifically, when students get stuck while solving programming problems, they need to receive next - step hints to guide them gradually towards the correct solution. However, existing methods are either too simplistic or not accurate enough to provide effective assistance.

To solve this problem, the paper proposes a new system that combines large - language models (LLMs) and static analysis techniques to generate high - quality text and code hints. The purpose of this system is to improve the quality of hints in the following ways:

1. **Subgoals Generation**: First, the system will analyze the task and generate a series of specific and manageable subgoals according to the current code state of the student. These subgoals break the task into smaller parts, ensuring that each hint is fine - grained, thus maintaining an appropriate level of challenge and support.

2. **Code Hint Generation**: Next, the system uses static analysis to control the quality of the generated code and ensure the appropriate size of the hints. Specific practices include:
- **Removing Irrelevant Changes**: Ignore modification suggestions for functions that the student has already implemented.
- **Handling Short Functions**: For very short functions, directly use the model solution provided by the task creator to avoid the LLM generating inaccurate code.
- **Ensuring Code Quality**: Use the code quality inspection tool of the IDE to optimize the code and correct common security and style issues.
- **Controlling Hint Size**: Through static analysis, ensure that each generated hint contains only one logical action, not multiple steps.

3. **Textual Hint Generation**: Finally, the system generates corresponding text descriptions based on the generated code hints to help students understand the specific operations that need to be performed. Textual hints are divided into two types:
- **Instrumental Help**: Inform students what to do next, presented in concise descriptive sentences.
- **Orientational Help**: Inform students which part of the code they should focus on by highlighting the location in the code editor.

Through this multi - stage approach, this system aims to provide more personalized and effective help, enabling students to better understand and complete programming tasks.