The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers

James Prather,Brent Reeves,Juho Leinonen,Stephen MacNeil,Arisoa S. Randrianasolo,Brett Becker,Bailey Kimmel,Jared Wright,Ben Briggs

2024-05-28

Abstract:Novice programmers often struggle through programming problem solving due to a lack of metacognitive awareness and strategies. Previous research has shown that novices can encounter multiple metacognitive difficulties while programming. Novices are typically unaware of how these difficulties are hindering their progress. Meanwhile, many novices are now programming with generative AI (GenAI), which can provide complete solutions to most introductory programming problems, code suggestions, hints for next steps when stuck, and explain cryptic error messages. Its impact on novice metacognition has only started to be explored. Here we replicate a previous study that examined novice programming problem solving behavior and extend it by incorporating GenAI tools. Through 21 lab sessions consisting of participant observation, interview, and eye tracking, we explore how novices are coding with GenAI tools. Although 20 of 21 students completed the assigned programming problem, our findings show an unfortunate divide in the use of GenAI tools between students who accelerated and students who struggled. Students who accelerated were able to use GenAI to create code they already intended to make and were able to ignore unhelpful or incorrect inline code suggestions. But for students who struggled, our findings indicate that previously known metacognitive difficulties persist, and that GenAI unfortunately can compound them and even introduce new metacognitive difficulties. Furthermore, struggling students often expressed cognitive dissonance about their problem solving ability, thought they performed better than they did, and finished with an illusion of competence. Based on our observations from both groups, we propose ways to scaffold the novice GenAI experience and make suggestions for future work.

Artificial Intelligence,Human-Computer Interaction

What problem does this paper attempt to address?

### Problems the Paper Attempts to Solve

The paper primarily explores the impact of Generative AI (GenAI) tools on novice programmers, including the benefits and potential harms they bring. Specifically:

1. **Research Background**:
- Programming education is undergoing transformation due to GenAI tools like ChatGPT and GitHub Copilot. These tools can address many programming-related tasks.
- Despite the powerful capabilities of GenAI, its impact on student learning remains unclear, especially at the beginner level, as most research focuses on professional developers.

2. **Core Issues**:
- The research aims to explore how GenAI tools affect the cognitive processes of novice programmers, particularly their metacognitive abilities.
- Metacognition refers to an individual's understanding and regulation of their cognitive processes, which is crucial in learning programming. Researchers hope to understand whether GenAI helps improve or rather undermines students' metacognitive abilities.

3. **Research Methods**:
- By replicating previous studies and incorporating GenAI tools, the study observes the problems students encounter during programming.
- Data is collected using eye-tracking technology and interviews to analyze students' behavior and thought processes while solving problems.

4. **Findings**:
- Results show that although most students can complete programming tasks, there is a significant polarization: some students effectively use GenAI to accelerate problem-solving, while others encounter more difficulties in the process.
- Students facing difficulties exhibit persistent metacognitive issues, and GenAI may exacerbate these problems or even introduce new challenges.
- Students with better academic performance and higher self-efficacy are more likely to use GenAI tools effectively; whereas, lower-performing students may fall into a "false sense of competence," believing they are performing better than they actually are.

In summary, the paper attempts to reveal the specific impacts of GenAI tools on novice programmers through experiments and data analysis, particularly focusing on the pros and cons in terms of metacognition.