Which is a better programming assistant? A comparative study between chatgpt and stack overflow

Jinrun Liu,Xinyu Tang,Linlin Li,Panpan Chen,Yepang Liu

2023-08-26

Abstract:Programmers often seek help from Q\&A websites to resolve issues they encounter during programming. Stack Overflow has been a widely used platform for this purpose for over a decade. Recently, revolutionary AI-powered platforms like ChatGPT have quickly gained popularity among programmers for their efficient and personalized programming assistance via natural language interactions. Both platforms can offer valuable assistance to programmers, but it's unclear which is more effective at enhancing programmer productivity. In our paper, we conducted an exploratory user study to compare the performance of Stack Overflow and ChatGPT in enhancing programmer productivity. Two groups of students with similar programming abilities were instructed to use the two platforms to solve three different types of programming tasks: algorithmic challenges, library usage, and debugging. During the experiments, we measured and compared the quality of code produced and the time taken to complete tasks for the two groups. The results show that, concerning code quality, ChatGPT outperforms Stack Overflow significantly in helping complete algorithmic and library-related tasks, while Stack Overflow is better for debugging tasks. Regarding task completion speed, the ChatGPT group is obviously faster than the Stack Overflow group in the algorithmic challenge, but the two groups have a similar performance in the other two tasks. Additionally, we conducted a post-experiment survey with the participants to understand how the platforms have helped them complete the programming tasks. We analyzed the questionnaires to summarize ChatGPT and Stack Overflow's strengths and weaknesses pointed out by the participants. By comparing these, we identified the reasons behind the two platforms' divergent performances in programming assistance.

Software Engineering

What problem does this paper attempt to address?

The paper aims to address the effectiveness of programming assistance tools (such as ChatGPT and Stack Overflow) in enhancing programmer productivity. Specifically, the study focuses on the following two research questions:

1. **RQ1: Which platform improves the code quality of programmers more?**
- The study compares the code quality performance of ChatGPT and Stack Overflow in algorithm tasks, library usage, and debugging tasks through experiments.

2. **RQ2: Which platform enhances the speed of task completion for programmers more?**
- The study also evaluates the completion speed of different programming tasks on these two platforms.

The research designed an experiment, inviting two groups of students with similar programming abilities to use ChatGPT and Stack Overflow respectively to complete three types of programming tasks: algorithm challenges, library usage, and debugging. During the experiment, the code quality and the time required to complete the tasks were recorded and analyzed in detail. The results show that in algorithm and library-related tasks, ChatGPT significantly outperformed Stack Overflow in terms of code quality; however, in debugging tasks, Stack Overflow performed better. Additionally, in the speed of algorithm challenge tasks, the ChatGPT group was significantly faster than the Stack Overflow group, but in the other two types of tasks, both groups performed similarly.