How Beginning Programmers and Code LLMs (Mis)read Each Other

Sydney Nguyen,Hannah McLean Babe,Yangtian Zi,Arjun Guha,Carolyn Jane Anderson,Molly Q Feldman

DOI: https://doi.org/10.1145/3613904.3642706

2024-07-08

Abstract:Generative AI models, specifically large language models (LLMs), have made strides towards the long-standing goal of text-to-code generation. This progress has invited numerous studies of user interaction. However, less is known about the struggles and strategies of non-experts, for whom each step of the text-to-code problem presents challenges: describing their intent in natural language, evaluating the correctness of generated code, and editing prompts when the generated code is incorrect. This paper presents a large-scale controlled study of how 120 beginning coders across three academic institutions approach writing and editing prompts. A novel experimental design allows us to target specific steps in the text-to-code process and reveals that beginners struggle with writing and editing prompts, even for problems at their skill level and when correctness is automatically determined. Our mixed-methods evaluation provides insight into student processes and perceptions with key implications for non-expert Code LLM use within and outside of education.

Human-Computer Interaction

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: the interaction problem between novice programmers and large - language models for code generation (Code LLMs) in the process of natural - language - to - code conversion. Specifically, the paper focuses on the challenges faced by non - expert users (especially novice programmers) when using Code LLMs and the strategies they adopt. These challenges include:

1. **Describing Intent**: How to clearly describe in natural language the program functions they want to achieve.
2. **Evaluating Code Quality**: How to judge whether the code generated by Code LLMs is correct.
3. **Modifying Prompts**: When the generated code is incorrect, how to effectively modify their natural - language prompts.

Through a large - scale controlled experiment, the paper studied the process of writing and editing prompts by 120 novice programmers in three academic institutions. The experimental design aims at specific steps in the text - to - code generation process, revealing the difficulties that novices encounter in writing and editing prompts, even when these problems are within their skill level and the correctness can be automatically determined. The research adopts a mixed - method evaluation, providing key insights into student processes and perceptions, and is of great significance for non - experts using Code LLMs both inside and outside education.