ChatGPTest: opportunities and cautionary tales of utilizing AI for questionnaire pretesting

Francisco Olivos,Minhui Liu
2024-05-10
Abstract:The rapid advancements in generative artificial intelligence have opened up new avenues for enhancing various aspects of research, including the design and evaluation of survey questionnaires. However, the recent pioneering applications have not considered questionnaire pretesting. This article explores the use of GPT models as a useful tool for pretesting survey questionnaires, particularly in the early stages of survey design. Illustrated with two applications, the article suggests incorporating GPT feedback as an additional stage before human pretesting, potentially reducing successive iterations. The article also emphasizes the indispensable role of researchers' judgment in interpreting and implementing AI-generated feedback.
Computers and Society,Artificial Intelligence
What problem does this paper attempt to address?
The paper attempts to address the issue of utilizing artificial intelligence (AI) technology, particularly generative AI (GPT models), in the pretesting process of questionnaires to improve the quality and efficiency of questionnaire design. Specifically, the paper explores how to incorporate an additional stage of GPT feedback before human pretesting, thereby reducing the number of subsequent iterations. Furthermore, the paper emphasizes the indispensable role of researchers in interpreting and implementing AI-generated feedback, noting that while AI can provide valuable suggestions, it ultimately requires the researchers' judgment to determine which feedback is appropriate. Through two specific application examples, the paper demonstrates the potential advantages and limitations of GPT models in questionnaire pretesting.