Abstract:Background: Manually analyzing public health-related content from social media provides valuable insights into the beliefs, attitudes, and behaviors of individuals, shedding light on trends and patterns that can inform public understanding, policy decisions, targeted interventions, and communication strategies. Unfortunately, the time and effort needed from well-trained human subject matter experts makes extensive manual social media listening unfeasible. Generative large language models (LLMs) can potentially summarize and interpret large amounts of text, but it is unclear to what extent LLMs can glean subtle health-related meanings in large sets of social media posts and reasonably report health-related themes. Objective: We aimed to assess the feasibility of using LLMs for topic model selection or inductive thematic analysis of large contents of social media posts by attempting to answer the following question: Can LLMs conduct topic model selection and inductive thematic analysis as effectively as humans did in a prior manual study, or at least reasonably, as judged by subject matter experts? Methods: We asked the same research question and used the same set of social media content for both the LLM selection of relevant topics and the LLM analysis of themes as was conducted manually in a published study about vaccine rhetoric. We used the results from that study as background for this LLM experiment by comparing the results from the prior manual human analyses with the analyses from 3 LLMs: GPT4-32K, Claude-instant-100K, and Claude-2-100K. We also assessed if multiple LLMs had equivalent ability and assessed the consistency of repeated analysis from each LLM. Results: The LLMs generally gave high rankings to the topics chosen previously by humans as most relevant. We reject a null hypothesis (P<.001, overall comparison) and conclude that these LLMs are more likely to include the human-rated top 5 content areas in their top rankings than would occur by chance. Regarding theme identification, LLMs identified several themes similar to those identified by humans, with very low hallucination rates. Variability occurred between LLMs and between test runs of an individual LLM. Despite not consistently matching the human-generated themes, subject matter experts found themes generated by the LLMs were still reasonable and relevant. Conclusions: LLMs can effectively and efficiently process large social media-based health-related data sets. LLMs can extract themes from such data that human subject matter experts deem reasonable. However, we were unable to show that the LLMs we tested can replicate the depth of analysis from human subject matter experts by consistently extracting the same themes from the same data. There is vast potential, once better validated, for automated LLM-based real-time social listening for common and rare health conditions, informing public health understanding of the public's interests and concerns and determining the public's ideas to address them.

Inductive reasoning with large language models: a simulated randomized controlled trial for epilepsy

SemioLLM: Assessing Large Language Models for Semiological Analysis in Epilepsy Research

Potential merits and flaws of large language models in epilepsy care: A critical review

Large Language Model Influence on Diagnostic Reasoning: A Randomized Clinical Trial

Implications of Large Language Models for Quality and Efficiency of Neurologic Care: Emerging Issues in Neurology

Large Language Model Influence on Diagnostic Reasoning

Accuracy, Consistency, and Hallucination of Large Language Models When Analyzing Unstructured Clinical Notes in Electronic Medical Records.

Simulate Scientific Reasoning with Multiple Large Language Models: An Application to Alzheimer's Disease Combinatorial Therapy

An Active Inference Strategy for Prompting Reliable Responses from Large Language Models in Medical Practice

Unlocking the potential of advanced large language models in medication review and reconciliation: A proof-of-concept investigation

Coupling Symbolic Reasoning with Language Modeling for Efficient Longitudinal Understanding of Unstructured Electronic Medical Records

Can Large Language Models Replace Data Scientists in Clinical Research?

Leveraging artificial intelligence to detect ethical concerns in medical research: a case study

Disparities in seizure outcomes revealed by large language models

Bridging the conversational gap in epilepsy: Using large language models to reveal insights into patient behavior and concerns from online discussions

Evaluating and Mitigating Limitations of Large Language Models in Clinical Decision Making

Large Language Model Prompting Techniques for Advancement in Clinical Medicine

Machine Learning and Artificial Intelligence Applications to Epilepsy: a Review for the Practicing Epileptologist

Large Language Model Influence on Management Reasoning: A Randomized Controlled Trial

Large Language Models Can Enable Inductive Thematic Analysis of a Social Media Corpus in a Single Prompt: Human Validation Study

Large Language Models Illuminate a Progressive Pathway to Artificial Intelligent Healthcare Assistant