Abstract:Background and Objectives: In pediatric oncology, caregivers seek detailed, accurate, and understandable information about their child's condition, treatment, and side effects. The primary aim of this study was to assess the performance of four publicly accessible large language model (LLM) supported knowledge generation and search tools in providing valuable and reliable information to caregivers of children with cancer. Methods: This cross-sectional study evaluated the performance of the four LLM-supported tools- ChatGPT (GPT-4), Google Bard (Gemini Pro), Microsoft Bing Chat, and Google SGE- against a set of frequently asked questions (FAQs) derived from the Children's Oncology Group Family Handbook and expert input. Five pediatric oncology experts assessed the generated LLM responses using measures including Accuracy (3-point ordinal scale), Clarity (3-point ordinal scale), Inclusivity (3-point ordinal scale), Completeness (Dichotomous nominal scale), Clinical Utility (5-point Likert-scale), and Overall Rating (4-point ordinal scale). Additional Content Quality Criteria such as Readability (ordinal scale; 5-18th grade of educated reading), Presence of AI Disclosure (Dichotomous scale), Source Credibility (3-point interval scale) , Resource Matching (3-point ordinal scale), and Content Originality (ratio scale) were also evaluated. We used descriptive analysis including the mean, standard deviation, median, and interquartile range. We conducted Shapiro-Wilk test for normality, Levene's test for homogeneity of variances, and Kruskal-Wallis H-Tests and Dunn's post-hoc tests for pairwise comparisons. Results: Through expert evaluation, ChatGPT showed high performance in accuracy (M=2.71, SD=0.235), clarity (M=2.73, SD=0.271), completeness (M=0.815, SD=0.203), Clinical Utility (M=3.81, SD=0.544), and Overall Rating (M=3.13, SD=0.419). Bard also performed well, especially in accuracy (M=2.56, SD=0.400) and clarity (M=2.54, SD=0.411), while Bing Chat (Accuracy M=2.33, SD=0.456; Clarity M=2.29, SD=0.424) and Google SGE (Accuracy M=2.08, SD=0.552; Clarity M=1.95, SD=0.541) had lower overall scores. The Presence of AI Disclosure was less frequent in ChatGPT (M=0.69, SD=0.46), which affected Clarity (M=2.73, SD=0.266), whereas Bard maintained a balance between AI Disclosure (M=0.92, SD=0.27) and Clarity (M=2.54, SD=0.403). Overall, we observed significant differences between LLM tools (p < .01). Conclusions: LLM-supported tools potentially contribute to caregivers' knowledge of pediatric oncology on related topics. Each model has unique strengths and areas for improvement, suggesting the need for careful selection and evaluation based on specific clinical contexts. Further research is needed to explore the application of these tools in other medical specialties and patient demographics to assess their broader applicability and long-term impacts, including the usability and feasibility of using LLM-supported tools with caregivers.

Proof-of-concept study of a small language model chatbot for breast cancer decision support – a transparent, source-controlled, explainable and data-secure approach

Evolution of publicly available large language models for complex decision-making in breast cancer care

Leveraging Large Language Models for Decision Support in Personalized Oncology

Integrating human expertise & automated methods for a dynamic and multi-parametric evaluation of large language models' feasibility in clinical decision-making

Large language model (ChatGPT) as a support tool for breast tumor board

Large language models for precision oncology: Clinical decision support through expert-guided learning.

A Basis for the Development of a Prospective Comparative Study

Large Language Model-Based Evaluation of Medical Question Answering Systems: Algorithm Development and Case Study

Large language models as decision aids in neuro-oncology: a review of shared decision-making applications

[Large Language Models for Rapid Simplification of Quality Assurance Data Input: Field Trial with Real Data in the Context of Tumour Documentation in Urology]

Applications of Large Language Models (LLMs) in Breast Cancer Care

Systematic analysis of ChatGPT, Google search and Llama 2 for clinical decision support tasks

Exploring Large Language Models for Specialist-level Oncology Care

Evaluating and Mitigating Limitations of Large Language Models in Clinical Decision Making

Exploring the role of artificial intelligence, large language models: Comparing patient‐focused information and clinical decision support capabilities to the gynecologic oncology guidelines

Clinical Camel: An Open-Source Expert-Level Medical Language Model with Dialogue-Based Knowledge Encoding

Can Large Language Models Aid Caregivers of Pediatric Cancer Patients in Information Seeking? A Cross-Sectional Investigation

Effects of interacting with a large language model compared with a human coach on the clinical diagnostic process and outcomes among fourth-year medical students: study protocol for a prospective, randomised experiment using patient vignettes

Autonomous Artificial Intelligence Agents for Clinical Decision Making in Oncology

Evaluation and mitigation of the limitations of large language models in clinical decision-making

Development of a privacy preserving large language model for automated data extraction from thyroid cancer pathology reports