Evaluating Language Models for Generating and Judging Programming Feedback

Charles Koutcheme,Nicola Dainese,Arto Hellas,Sami Sarsa,Juho Leinonen,Syed Ashraf,Paul Denny

2024-07-06

Abstract:The emergence of large language models (LLMs) has transformed research and practice in a wide range of domains. Within the computing education research (CER) domain, LLMs have received plenty of attention especially in the context of learning programming. Much of the work on LLMs in CER has however focused on applying and evaluating proprietary models. In this article, we evaluate the efficiency of open-source LLMs in generating high-quality feedback for programming assignments, and in judging the quality of the programming feedback, contrasting the results against proprietary models. Our evaluations on a dataset of students' submissions to Python introductory programming exercises suggest that the state-of-the-art open-source LLMs (Meta's Llama3) are almost on-par with proprietary models (GPT-4o) in both the generation and assessment of programming feedback. We further demonstrate the efficiency of smaller LLMs in the tasks, and highlight that there are a wide range of LLMs that are accessible even for free for educators and practitioners.

Artificial Intelligence,Computers and Society

What problem does this paper attempt to address?

The paper aims to address the following issues:

1. **Comparison of Open Source and Proprietary Models in Generating Programming Feedback**: The study evaluates the efficiency of open-source large-scale language models (such as Meta's Llama3) versus proprietary models (such as GPT-4) in generating high-quality programming assignment feedback and compares these models' ability to judge the quality of programming feedback.

2. **Automatic Evaluation of Programming Feedback Quality**: The paper explores the capability of open-source and proprietary models as "judges," i.e., whether they can effectively evaluate the quality of programming feedback generated by other models and compares their results with expert human judgments.

Experiments reveal that open-source models can almost match proprietary models in generating and evaluating programming feedback, especially for smaller open-source models, which also perform exceptionally well. Additionally, the paper discusses how to use language models to automatically evaluate feedback quality in the absence of reference answers and proposes using a "jury" of multiple models to improve evaluation accuracy. These findings are significant for educators and practitioners as they can obtain efficient programming teaching support without relying on expensive proprietary tools.