Abstract:Context: The advent of AI-driven large language models (LLMs), such as ChatGPT 3.5 and GPT-4, have stirred discussions about their role in qualitative research. Some view these as tools to enrich human understanding, while others perceive them as threats to the core values of the discipline. Problem: A significant concern revolves around the disparity between AI-generated classifications and human comprehension, prompting questions about the reliability of AI-derived insights. An “AI echo chamber” could potentially risk the diversity inherent in qualitative research. A minimal overlap between AI and human interpretations amplifies concerns about the fading human element in research. Objective: This study aimed to compare and contrast the comprehension capabilities of humans and LLMs, specifically ChatGPT 3.5 and GPT-4. Methodology: We conducted an experiment with small sample of Alexa app reviews, initially classified by a human analyst. ChatGPT 3.5 and GPT-4 were then asked to classify these reviews and provide the reasoning behind each classification. We compared the results with human classification and reasoning. Results: The research indicated a significant alignment between human and ChatGPT 3.5 classifications in one-third of cases, and a slightly lower alignment with GPT-4 in over a quarter of cases. The two AI models showed a higher alignment, observed in more than half of the instances. However, a consensus across all three methods was seen only in about one-fifth of the classifications. In the comparison of human and LLMs reasoning, it appears that human analysts lean heavily on their individual experiences. As expected, LLMs, on the other hand, base their reasoning on the specific word choices found in app reviews and the functional components of the app itself. Conclusion: Our results highlight the potential for effective human-LLM collaboration, suggesting a synergistic rather than competitive relationship. Researchers must continuously evaluate LLMs’ role in their work, thereby fostering a future where AI and humans jointly enrich qualitative research.

Can AI serve as a substitute for human subjects in software engineering research?

Can AI Serve as a Substitute for Human Subjects in Software Engineering Research?

Can AI language models replace human participants?

Human-AI Collaboration in Thematic Analysis using ChatGPT: A User Study and Design Recommendations

The illusion of artificial inclusion

Towards Integrating Emerging AI Applications in SE Education

The Ethics of Using Artificial Intelligence in Qualitative Research

Comparing Software Developers with ChatGPT: An Empirical Investigation

AI and Human Reasoning: Qualitative Research in the Age of Large Language Models

Human AI Collaboration in Software Engineering: Lessons Learned from a Hands On Workshop

Supporting Serendipity: Opportunities and Challenges for Human-AI Collaboration in Qualitative Analysis

Can linguists distinguish between ChatGPT/AI and human writing?: A study of research ethics and academic publishing

AI Conversational Interviewing: Transforming Surveys with LLMs as Adaptive Interviewers

Empowering Qualitative Research Methods in Education with Artificial Intelligence

Artificial intelligence in psychology research

Machine-assisted quantitizing designs: augmenting humanities and social sciences with artificial intelligence

Towards Applying Powerful Large AI Models in Classroom Teaching: Opportunities, Challenges and Prospects

Elicit: AI literature review research assistant

Artificial intelligence in applied (linguistics): a content analysis and future prospects

Enhancing UX Research Activities Using GenAI -- Potential Applications and Challenges

Can Large Language Models Serve as Data Analysts? A Multi-Agent Assisted Approach for Qualitative Data Analysis