Abstract:This paper aims to answer one central question: to what extent can open-source generative text models be used in a workflow to approximate thematic analysis in social science research? To answer this question, we present the Generative AI-enabled Theme Organization and Structuring (GATOS) workflow, which uses open-source machine learning techniques, natural language processing tools, and generative text models to facilitate thematic analysis. To establish validity of the method, we present three case studies applying the GATOS workflow, leveraging these models and techniques to inductively create codebooks similar to traditional procedures using thematic analysis. Specifically, we investigate the extent to which a workflow comprising open-source models and tools can inductively produce codebooks that approach the known space of themes and sub-themes. To address the challenge of gleaning insights from these texts, we combine open-source generative text models, retrieval-augmented generation, and prompt engineering to identify codes and themes in large volumes of text, i.e., generate a qualitative codebook. The process mimics an inductive coding process that researchers might use in traditional thematic analysis by reading text one unit of analysis at a time, considering existing codes already in the codebook, and then deciding whether or not to generate a new code based on whether the extant codebook provides adequate thematic coverage. We demonstrate this workflow using three synthetic datasets from hypothetical organizational research settings: a study of teammate feedback in teamwork settings, a study of organizational cultures of ethical behavior, and a study of employee perspectives about returning to their offices after the pandemic. We show that the GATOS workflow is able to identify themes in the text that were used to generate the original synthetic datasets.

What problem does this paper attempt to address?

The paper attempts to address the issue of how to approximate thematic analysis in social science research using open-source generative text models and machine learning techniques. Specifically, a major challenge faced by researchers is that traditional manual coding methods are time-consuming, labor-intensive, and difficult to scale when dealing with large volumes of text data. For example, hundreds of hours of audio recordings, thousands of written documents, and responses to open-ended survey questions may contain important insights into phenomena, but the sheer volume of data makes manual analysis impractical. Therefore, this paper proposes a workflow named GATOS (Generative AI-enabled Theme Organization and Structuring), which aims to combine open-source machine learning technologies, natural language processing tools, and generative text models to generate qualitative codebooks in an automated or semi-automated manner. This approach effectively identifies themes and sub-themes from large volumes of text data, improving analysis efficiency while maintaining the quality and accuracy consistent with traditional thematic analysis methods.

Thematic Analysis with Open-Source Generative AI and Machine Learning: A New Method for Inductive Qualitative Codebook Development

Using ChatGPT for Thematic Analysis

Attempting rigour and replicability in thematic analysis of qualitative research data; a case study of codebook development

A Team-based Approach to Open Coding: Considerations for Creating Intercoder Consensus

A Computational Method for Measuring "Open Codes" in Qualitative Analysis

Harnessing ChatGPT for Thematic Analysis: Are We Ready?

Performing an Inductive Thematic Analysis of Semi-Structured Interviews With a Large Language Model: An Exploration and Provocation on the Limits of the Approach

Exploring the Use of Artificial Intelligence for Qualitative Data Analysis: The Case of ChatGPT

Using Large Language Models to Support Thematic Analysis in Empirical Legal Studies

Can Large Language Models emulate an inductive Thematic Analysis of semi-structured interviews? An exploration and provocation on the limits of the approach and the model

Template Analysis in Business and Management Research

Redefining Qualitative Analysis in the AI Era: Utilizing ChatGPT for Efficient Thematic Analysis

Inductive thematic analysis of healthcare qualitative interviews using open-source large language models: How does it compare to traditional methods?

Prompts Matter: Comparing ML/GAI Approaches for Generating Inductive Qualitative Coding Results

Human-AI Collaboration in Thematic Analysis using ChatGPT: A User Study and Design Recommendations

Conceptual and design thinking for thematic analysis

Analyzing Open-ended Questions by Means of Text Analysis Procedures

Using Generative Text Models to Create Qualitative Codebooks for Student Evaluations of Teaching

Towards Coding Social Science Datasets with Language Models

How generative AI models such as ChatGPT can be (mis)used in SPC practice, education, and research? An exploratory study

Reflections on Inductive Thematic Saturation as a potential metric for measuring the validity of an inductive Thematic Analysis with LLMs