Abstract:In the current era of artificial intelligence, large language models such as ChatGPT and BARD are being increasingly used for various applications, such as language translation, text generation, and human-like conversation. The fact that these models consist of large amounts of data, including many different opinions and perspectives, could introduce the possibility of a new qualitative research approach: Due to the probabilistic character of their answers, “interviewing” these large language models could give insights into public opinions in a way that otherwise only interviews with large groups of subjects could deliver. However, it is not yet clear if qualitative content analysis research methods can be applied to interviews with these models. Evaluating the applicability of qualitative research methods to interviews with large language models could foster our understanding of their abilities and limitations. In this paper, we examine the applicability of qualitative content analysis research methods to interviews with ChatGPT in English, ChatGPT in German, and BARD in English on the relevance of computer science in K-12 education, which was used as an exemplary topic. We found that the answers produced by these models strongly depended on the provided context, and the same model could produce heavily differing results for the same questions. From these results and the insights throughout the process, we formulated guidelines for conducting and analyzing interviews with large language models. Our findings suggest that qualitative content analysis research methods can indeed be applied to interviews with large language models, but with careful consideration of contextual factors that may affect the responses produced by these models. The guidelines we provide can aid researchers and practitioners in conducting more nuanced and insightful interviews with large language models. From an overall view of our results, we generally do not recommend using interviews with large language models for research purposes, due to their highly unpredictable results. However, we suggest using these models as exploration tools for gaining different perspectives on research topics and for testing interview guidelines before conducting real-world interviews.

Enhancing qualitative research in psychology with large language models: a methodological exploration and examples of simulations

When Qualitative Research Meets Large Language Model: Exploring the Potential of QualiGPT as a Tool for Qualitative Coding

Perils and opportunities in using large language models in psychological research

Large Language Model for Qualitative Research -- A Systematic Mapping Study

Apprentices to Research Assistants: Advancing Research with Large Language Models

Large Language Models in Qualitative Research: Can We Do the Data Justice?

'Simulacrum of Stories': Examining Large Language Models as Qualitative Research Participants

Exploring the Frontiers of LLMs in Psychological Applications: A Comprehensive Review

Using large language models in psychology

Qualitative Research Methods for Large Language Models: Conducting Semi-Structured Interviews with ChatGPT and BARD on Computer Science Education

Exploring the use of a Large Language Model for data extraction in systematic reviews: a rapid feasibility study

Evaluating Large Language Models in Analysing Classroom Dialogue

Exploring Qualitative Research Using LLMs

GPT-Based Models Meet Simulation: How to Efficiently Use Large-Scale Pre-Trained Language Models Across Simulation Tasks

Evaluating Research Quality with Large Language Models: An Analysis of ChatGPT's Effectiveness with Different Settings and Inputs

Applications and Implications of Large Language Models in Qualitative Analysis: A New Frontier for Empirical Software Engineering

Artificial intelligence and qualitative research: The promise and perils of large language model (LLM) ‘assistance’

Framework-based qualitative analysis of free responses of Large Language Models: Algorithmic fidelity

An Examination of the Use of Large Language Models to Aid Analysis of Textual Data