Abstract:Abstract Large language models (LLMs) such as ChatGPT can imitate human conversation and produce rapid, coherent responses, which may mask their potential for inaccuracies. With patients increasingly turning to the internet for medical information, the use of LLM chatbots for cancer-related queries risks spreading misinformation. Our study assessed ChatGPT’s accuracy and reproducibility in offering valid information and treatment advice for lung cancer in line with established guidelines. In the evolving landscape of AI-driven healthcare support, the ability of language models to provide accurate and reliable information is crucial. Our study delves into the effectiveness of OpenAI's ChatGPT models (versions 3.5 and 4.0) in responding to patient inquiries about lung cancer across various domains including general information, clinical presentation, risk factors, screening, diagnosis, staging, treatment options, prognosis, post-treatment follow-up, lifestyle recommendations, and psychosocial/educational aspects. We conducted a structured assessment, posing identical sets of questions to both ChatGPT 3.5 and 4.0. A total of 47 questions were posed with each query being repeated twice per model to evaluate both the accuracy and reproducibility of the responses. The scoring system focused on the accuracy and comprehensiveness of each response. Our findings revealed a notable disparity in the performance of the two models. GPT 4.0 demonstrated higher consistency and accuracy, with 41 out of 47 (87.2%) responses deemed accurate and comprehensive, compared to 36 out of 47 (76.6%) for GPT 3.5. In terms of reproducibility, both models exhibited strong performance: 42 out of 47 (89.3%) for GPT 3.5 and 45 out of 47 for GPT 4.0 (95.7%). When comparing responses between the models, we observed good reproducibility in 38 out of 47 questions (80.8%). A key observation was that GPT 4.0 significantly outperformed its predecessor GPT 3.5 in terms of both accuracy as well as reproducibility within its own responses, indicating a more reliable and consistent performance. The area most lacking in accuracy for both models was lung cancer staging, indicating a need for further refinement in this domain. Another key observation was the models' tendency to incorporate empathetic language, often beginning responses with expressions of sympathy and consistently advising confirmation with a medical professional. Our study underscores the potential and limitations of current AI models in patient education and support, highlighting areas for improvement and the importance of empathetic communication in AI interactions with patients. As the model continues to be trained on a larger and more comprehensive set of data, it is reasonable to anticipate further improvements in its ability to provide precise, detailed, and contextually appropriate responses. Citation Format: Asiyah Allibhai, Ahmed Allibhai, Anthony Brade, Zishan Allibhai. Evaluating the accuracy and reproducibility of ChatGPT models in answering lung cancer patient queries [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2024; Part 1 (Regular Abstracts); 2024 Apr 5-10; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2024;84(6_Suppl):Abstract nr 1296.

Clinical decision making by ChatGPT vs medical oncologists: A retrospective concordance study.

Evaluation of ChatGPT-4's Performance in Therapeutic Decision-Making During Multidisciplinary Oncology Meetings for Head and Neck Squamous Cell Carcinoma

Assessing ChatGPT's potential as a clinical resource for medical oncologists: An evaluation with board-style questions and real-world patient cases.

Exploring the landscape of AI-assisted decision-making in head and neck cancer treatment: a comparative analysis of NCCN guidelines and ChatGPT responses

ChatGPT v4 outperforming v3.5 on cancer treatment recommendations in quality, clinical guideline, and expert opinion concordance

Implementing large language model-based artificial intelligence (AI) technology in proposing effective treatment plans in patients with cancer.

Decision-making concordance: A comparative analysis between a multidisciplinary tumor board and a large language model (ChatGPT) in breast cancer management.

Comparative analysis of ChatGPT-4.0's management of six gastrointestinal cancers according to the NCCN guidelines.

Assessing the role of advanced artificial intelligence as a tool in multidisciplinary tumor board decision-making for recurrent/metastatic head and neck cancer cases - the first study on ChatGPT 4o and a comparison to ChatGPT 4.0

Comparative effectiveness of ChatGPT 4.0 and medical oncologists in telemedicine-based management of metastatic prostate carcinoma (mPC).

OncoGPT: Enhancing oncological decision-making with ASCO guideline–trained AI.

Abstract 1296: Evaluating the accuracy and reproducibility of ChatGPT models in answering lung cancer patient queries

Assessing Generative Pretrained Transformers (GPT) in Clinical Decision-Making: Comparative Analysis of GPT-3.5 and GPT-4

Exploring the role of ChatGPT in clinical decision-making in otorhinolaryngology: a ChatGPT designed study

Are Different Versions of ChatGPT's Ability Comparable to the Clinical Diagnosis Presented in Case Reports? A Descriptive Study

ChatGPT in Iranian medical licensing examination: evaluating the diagnostic accuracy and decision-making capabilities of an AI-based model

Assessing the role of advanced artificial intelligence as a tool in multidisciplinary tumor board decision-making for primary head and neck cancer cases

Advancing Medical Practice with Artificial Intelligence: ChatGPT in Healthcare

Effectiveness of ChatGPT 4.0 in Telemedicine-Based Management of Metastatic Prostate Carcinoma

Evaluating ChatGPT as an adjunct for the multidisciplinary tumor board decision-making in primary breast cancer cases

A Comparison of ChatGPT and Multidisciplinary Team Meeting Treatment Recommendations in 10 Consecutive Cervical Cancer Patients