Abstract:This paper aims to answer one central question: to what extent can open-source generative text models be used in a workflow to approximate thematic analysis in social science research? To answer this question, we present the Generative AI-enabled Theme Organization and Structuring (GATOS) workflow, which uses open-source machine learning techniques, natural language processing tools, and generative text models to facilitate thematic analysis. To establish validity of the method, we present three case studies applying the GATOS workflow, leveraging these models and techniques to inductively create codebooks similar to traditional procedures using thematic analysis. Specifically, we investigate the extent to which a workflow comprising open-source models and tools can inductively produce codebooks that approach the known space of themes and sub-themes. To address the challenge of gleaning insights from these texts, we combine open-source generative text models, retrieval-augmented generation, and prompt engineering to identify codes and themes in large volumes of text, i.e., generate a qualitative codebook. The process mimics an inductive coding process that researchers might use in traditional thematic analysis by reading text one unit of analysis at a time, considering existing codes already in the codebook, and then deciding whether or not to generate a new code based on whether the extant codebook provides adequate thematic coverage. We demonstrate this workflow using three synthetic datasets from hypothetical organizational research settings: a study of teammate feedback in teamwork settings, a study of organizational cultures of ethical behavior, and a study of employee perspectives about returning to their offices after the pandemic. We show that the GATOS workflow is able to identify themes in the text that were used to generate the original synthetic datasets.

A Team-based Approach to Open Coding: Considerations for Creating Intercoder Consensus

Beyond a coefficient: an interactive process for achieving inter-rater consistency in qualitative coding

A Computational Method for Measuring "Open Codes" in Qualitative Analysis

Thematic Analysis with Open-Source Generative AI and Machine Learning: A New Method for Inductive Qualitative Codebook Development

Reliability in Software Engineering Qualitative Research through Inter-Coder Agreement: A guide using Krippendorff's $α$ & Atlas.ti

Data Analysis Methods for Qualitative Research: Managing the Challenges of Coding, Interrater Reliability, and Thematic Analysis

Ease on Down the Code: Complex Collaborative Qualitative Coding Simplified with 'Code Wizard'

Intercoder Reliability in Qualitative Research: Debates and Practical Guidelines

Intercoder Reliability for Use in Qualitative Research and Evaluation

CoAIcoder: Examining the Effectiveness of AI-assisted Human-to-Human Collaboration in Qualitative Analysis

The second step in data analysis: Coding qualitative research data

Impact of Human-AI Interaction on User Trust and Reliance in AI-Assisted Qualitative Coding

Attempting rigour and replicability in thematic analysis of qualitative research data; a case study of codebook development

Building Transparency and Trustworthiness in Inductive Research Through Computer-Aided Qualitative Data Analysis Software

Collaborative Codebook Development Across Information Worlds: Intersections Between Pedagogy, Theory, and Method in Three Dissertations

A Collaborative Way of Knowing: Bridging Computational Communication Research and Grounded Theory Ethnography

Engaging the Senses in Qualitative Research via Multimodal Coding: Triangulating Transcript, Audio, and Video Data in a Study With Sexual and Gender Minority Youth

Inter-Rater Reliability Methods in Qualitative Case Study Research

The Future of Coding: A Comparison of Hand-Coding and Three Types of Computer-Assisted Text Analysis Methods

Prompts Matter: Comparing ML/GAI Approaches for Generating Inductive Qualitative Coding Results

Analyzing Open-ended Questions by Means of Text Analysis Procedures