Computer Science Education in ChatGPT Era: Experiences from an Experiment in a Programming Course for Novice Programmers

Tomaž Kosar,Dragana Ostojić,Yu David Liu,Marjan Mernik

DOI: https://doi.org/10.3390/math12050629

IF: 2.4

2024-02-21

Mathematics

Abstract:The use of large language models with chatbots like ChatGPT has become increasingly popular among students, especially in Computer Science education. However, significant debates exist in the education community on the role of ChatGPT in learning. Therefore, it is critical to understand the potential impact of ChatGPT on the learning, engagement, and overall success of students in classrooms. In this empirical study, we report on a controlled experiment with 182 participants in a first-year undergraduate course on object-oriented programming. Our differential study divided students into two groups, one using ChatGPT and the other not using it for practical programming assignments. The study results showed that the students' performance is not influenced by ChatGPT usage (no statistical significance between groups with a p-value of 0.730), nor are the grading results of practical assignments (p-value 0.760) and midterm exams (p-value 0.856). Our findings from the controlled experiment suggest that it is safe for novice programmers to use ChatGPT if specific measures and adjustments are adopted in the education process.

mathematics

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the impact of using large - language models (such as ChatGPT) on students' learning, engagement, and overall performance in computer science education in the ChatGPT era. Specifically, the researchers hope to explore the following questions through a controlled experiment:

- Does using ChatGPT affect students' performance in practical assignments and mid - term exams?
- Does using ChatGPT affect students' overall performance in an introductory programming course?
- What is the impact of using ChatGPT on the final course grades?
- What are the purposes of students using ChatGPT in Programming II course?
- According to students' opinions, is ChatGPT helpful for learning programming?

To answer these questions, the researchers designed a controlled experiment. They divided 182 first - year undergraduates into two groups. One group was allowed to use ChatGPT, while the other was not. By comparing the performance of the two groups of students in practical assignments, mid - term exams, and final grades, the researchers hope to understand the actual role of ChatGPT in computer science education and its possible impacts.