Evaluating Privacy Questions From Stack Overflow: Can ChatGPT Compete?

Zack Delile,Sean Radel,Joe Godinez,Garrett Engstrom,Theo Brucker,Kenzie Young,Sepideh Ghanavati

DOI: https://doi.org/10.1109/REW57809.2023.00048

2023-06-20

Abstract:Stack Overflow and other similar forums are used commonly by developers to seek answers for their software development as well as privacy-related concerns. Recently, ChatGPT has been used as an alternative to generate code or produce responses to developers' questions. In this paper, we aim to understand developers' privacy challenges by evaluating the types of privacy-related questions asked on Stack Overflow. We then conduct a comparative analysis between the accepted responses given by Stack Overflow users and the responses produced by ChatGPT for those extracted questions to identify if ChatGPT could serve as a viable alternative. Our results show that most privacy-related questions are related to choice/consent, aggregation, and identification. Furthermore, our findings illustrate that ChatGPT generates similarly correct responses for about 56% of questions, while for the rest of the responses, the answers from Stack Overflow are slightly more accurate than ChatGPT.

Software Engineering,Artificial Intelligence,Computation and Language

What problem does this paper attempt to address?

The paper aims to address the following issues:

1. **Understanding the types of privacy-related questions developers ask on Stack Overflow**: By analyzing questions on Stack Overflow, identify the specific privacy challenges developers encounter during software development.
2. **Evaluating ChatGPT's performance in answering privacy-related questions**: Compare the answers generated by ChatGPT with those provided by Stack Overflow users to determine if ChatGPT can serve as an alternative for answering these privacy-related questions.

Specifically, researchers extracted 932 questions from Stack Overflow data between 2016 and 2023 and randomly selected 270 pairs of questions and their answers for annotation. Through multiple rounds of annotation and discussion, 92 privacy-related question-answer pairs were finally identified. The study found that most privacy questions focused on aspects such as choice/consent, aggregation, and identification. Additionally, the research indicated that for approximately 56% of the questions, ChatGPT was able to generate responses that matched the accepted answers on Stack Overflow. However, in the remaining cases, the answers on Stack Overflow were slightly more accurate. This suggests that while ChatGPT can serve as an alternative to Stack Overflow in some aspects, developers should still use these tools cautiously to ensure compliance with regulations and protect user privacy, as the accuracy has not yet reached 75%.