Can ChatGPT Pass An Introductory Level Functional Language Programming Course?

Chuqin Geng,Yihan Zhang,Brigitte Pientka,Xujie Si
2023-05-04
Abstract:The recent introduction of ChatGPT has drawn significant attention from both industry and academia due to its impressive capabilities in solving a diverse range of tasks, including language translation, text summarization, and computer programming. Its capability for writing, modifying, and even correcting code together with its ease of use and access is already dramatically impacting computer science education. This paper aims to explore how well ChatGPT can perform in an introductory-level functional language programming course. In our systematic evaluation, we treated ChatGPT as one of our students and demonstrated that it can achieve a grade B- and its rank in the class is 155 out of 314 students overall. Our comprehensive evaluation provides valuable insights into ChatGPT's impact from both student and instructor perspectives. Additionally, we identify several potential benefits that ChatGPT can offer to both groups. Overall, we believe that this study significantly clarifies and advances our understanding of ChatGPT's capabilities and potential impact on computer science education.
Computers and Society,Artificial Intelligence
What problem does this paper attempt to address?
This paper aims to explore the performance of ChatGPT in a first-year university functional programming course. Researchers treated ChatGPT as a student and systematically evaluated its ability to complete assignments and exams. The results showed that, without assistance, ChatGPT could achieve a grade equivalent to a B- (approximately 67%), ranking 155th out of 314 students. Additionally, by employing "assistance" strategies such as natural language prompts, example-based teaching, and providing test cases, researchers further improved ChatGPT's performance, raising its rank to 155th. These findings not only reveal the potential of ChatGPT in programming tasks but also provide insights into how it can be used as an educational tool. Overall, this study helps us better understand ChatGPT's capabilities and its potential impact on computer science education.