Abstract:Background: Data analysis approaches such as qualitative content analysis are notoriously time and labor intensive because of the time to detect, assess, and code a large amount of data. Tools such as ChatGPT may have tremendous potential in automating at least some of the analysis. Objective: The aim of this study was to explore the utility of ChatGPT in conducting qualitative content analysis through the analysis of forum posts from people sharing their experiences on reducing their sugar consumption. Methods: Inductive and deductive content analysis were performed on 537 forum posts to detect mechanisms of behavior change. Thorough prompt engineering provided appropriate instructions for ChatGPT to execute data analysis tasks. Data identification involved extracting change mechanisms from a subset of forum posts. The precision of the extracted data was assessed through comparison with human coding. On the basis of the identified change mechanisms, coding schemes were developed with ChatGPT using data-driven (inductive) and theory-driven (deductive) content analysis approaches. The deductive approach was informed by the Theoretical Domains Framework using both an unconstrained coding scheme and a structured coding matrix. In total, 10 coding schemes were created from a subset of data and then applied to the full data set in 10 new conversations, resulting in 100 conversations each for inductive and unconstrained deductive analysis. A total of 10 further conversations coded the full data set into the structured coding matrix. Intercoder agreement was evaluated across and within coding schemes. ChatGPT output was also evaluated by the researchers to assess whether it reflected prompt instructions. Results: The precision of detecting change mechanisms in the data subset ranged from 66% to 88%. Overall κ scores for intercoder agreement ranged from 0.72 to 0.82 across inductive coding schemes and from 0.58 to 0.73 across unconstrained coding schemes and structured coding matrix. Coding into the best-performing coding scheme resulted in category-specific κ scores ranging from 0.67 to 0.95 for the inductive approach and from 0.13 to 0.87 for the deductive approaches. ChatGPT largely followed prompt instructions in producing a description of each coding scheme, although the wording for the inductively developed coding schemes was lengthier than specified. Conclusions: ChatGPT appears fairly reliable in assisting with qualitative analysis. ChatGPT performed better in developing an inductive coding scheme that emerged from the data than adapting an existing framework into an unconstrained coding scheme or coding directly into a structured matrix. The potential for ChatGPT to act as a second coder also appears promising, with almost perfect agreement in at least 1 coding scheme. The findings suggest that ChatGPT could prove useful as a tool to assist in each phase of qualitative content analysis, but multiple iterations are required to determine the reliability of each stage of analysis.

Unravelling ChatGPT's potential in summarising qualitative in-depth interviews

ChatGPT Application In Summarizing An Evolution Of Deep Learning Techniques In Imaging: A Qualitative Study

Prompts, Pearls, Imperfections: Comparing ChatGPT and a Human Researcher in Qualitative Data Analysis

Using ChatGPT for Thematic Analysis

Harnessing ChatGPT for Thematic Analysis: Are We Ready?

Chat Generative Pre-Trained Transformer (ChatGPT): Boon or Bane – A Qualitative Meta-Synthesis

ChatGPT in forensic sciences: a new Pandora's box with advantages and challenges to pay attention

Redefining Qualitative Analysis in the AI Era: Utilizing ChatGPT for Efficient Thematic Analysis

Exploring the Limits of ChatGPT for Query or Aspect-based Text Summarization

Human-AI Collaboration in Thematic Analysis using ChatGPT: A User Study and Design Recommendations

ChatGPT for Automated Qualitative Research: Content Analysis

Human-like Summarization Evaluation with ChatGPT

Leveraging ChatGPT for Effective Research Design and Data Analysis — An Example with Research on Short Video Addiction

Revolutionizing Research in Social Sciences: The Transformative Potential of ChatGPT

Decoding ChatGPT: A Taxonomy of Existing Research, Current Challenges, and Possible Future Directions

Transforming Conversations with AI—A Comprehensive Study of ChatGPT

Generative AI as a Tool for Environmental Health Research Translation

Uncovering the Potential of ChatGPT for Discourse Analysis in Dialogue: An Empirical Study

Unleashing efficiency and insights: Exploring the potential applications and challenges of ChatGPT in accounting

Unlocking the Potential of ChatGPT: A Comprehensive Exploration of its Applications, Advantages, Limitations, and Future Directions in Natural Language Processing

Analysis of the Use of GPT Chat to Improve Student Performance