Assessment of the Reliability and Clinical Applicability of ChatGPT's Responses to Patients' Common Queries About Rosacea
Sihan Yan,Dan Du,Xu Liu,Yingying Dai,Min-Kyu Kim,Xinyu Zhou,Lian Wang,Lu Zhang,Xian Jiang
DOI: https://doi.org/10.2147/ppa.s444928
2024-02-01
Patient Preference and Adherence
Abstract:Sihan Yan, 1, 2, &ast Dan Du, 1, 2, &ast Xu Liu, 1, 2 Yingying Dai, 1, 2 Min-Kyu Kim, 1, 2 Xinyu Zhou, 3 Lian Wang, 1, 2 Lu Zhang, 1, 2 Xian Jiang 1, 2 1 Department of Dermatology, West China Hospital, Sichuan University, Chengdu, People's Republic of China; 2 Laboratory of Dermatology, Clinical Institute of Inflammation and Immunology, Frontiers Science Center for Disease-Related Molecular Network, West China Hospital, Sichuan University, Chengdu, People's Republic of China; 3 Department of Dermatology, Nanbu County People's Hospital, Nanbu County, Nanchong, Sichuan, People's Republic of China &astThese authors contributed equally to this work Correspondence: Xian Jiang; Lu Zhang, Email ; Objective: Artificial intelligence chatbot, particularly ChatGPT (Chat Generative Pre-trained Transformer), is capable of analyzing human input and generating human-like responses, which shows its potential application in healthcare. People with rosacea often have questions about alleviating symptoms and daily skin-care, which is suitable for ChatGPT to response. This study aims to assess the reliability and clinical applicability of ChatGPT 3.5 in responding to patients' common queries about rosacea and to evaluate the extent of ChatGPT's coverage in dermatology resources. Methods: Based on a qualitative analysis of the literature on the queries from rosacea patients, we have extracted 20 questions of patients' greatest concerns, covering four main categories: treatment, triggers and diet, skincare, and special manifestations of rosacea. Each question was inputted into ChatGPT separately for three rounds of question-and-answer conversations. The generated answers will be evaluated by three experienced dermatologists with postgraduate degrees and over five years of clinical experience in dermatology, to assess their reliability and applicability for clinical practice. Results: The analysis results indicate that the reviewers unanimously agreed that ChatGPT achieved a high reliability of 92.22% to 97.78% in responding to patients' common queries about rosacea. Additionally, almost all answers were applicable for supporting rosacea patient education, with a clinical applicability ranging from 98.61% to 100.00%. The consistency of the expert ratings was excellent (all significance levels were less than 0.05), with a consistency coefficient of 0.404 for content reliability and 0.456 for clinical practicality, indicating significant consistency in the results and a high level of agreement among the expert ratings. Conclusion: ChatGPT 3.5 exhibits excellent reliability and clinical applicability in responding to patients' common queries about rosacea. This artificial intelligence tool is applicable for supporting rosacea patient education. Keywords: artificial intelligence, ChatGPT, rosacea, patient education In the last decade, deep learning (DL) and other artificial intelligence (AI) technologies have made rapid progress. 1 Based on novel AI techniques, Augello et al developed a social chatbot model capable of integrating both individual and social processes. 2 This pioneering approach inspired researchers to dig out the potentials of AI chatbots, especially in the area of healthcare. Chung et al introduced a chatbot-based application, which aimed to provide timely assistance to these patients with chronic diseases. 3 AI chatbots can provide numerous advantages, including enhanced patient engagement, improved diagnostic accuracy, and personalized treatment plans and facilitating the assimilation of the latest medical literature into clinical practice. 4 As powerful as AI chatbots can be, there are also concerns about the limitations of them being medical chatbots, including the accuracy and reliability of the medical information they provide, the transparency of the model, and the ethics of making use of user information. 5 Chat Generative Pre-trained Transformer (ChatGPT) series is a highly discussed one of those artificial intelligence tools developed by OpenAI. It is based on natural language processing (NLP) models. It utilizes deep learning to provide coherent and fluent human-like text responses to complex queries. Other significant AI chatbots, such as Google Med-PaLM are also being explored for medical applications. Also using chatbot developing tools, including IBM Watson (IBM Corp) assistant, some researchers are constructing other specialized AI chatbots focused on certain specified fields of medicine. 6,7 In certain fields of medici -Abstract Truncated-
medicine, general & internal