Abstract:Importance: Although augmenting large language models (LLMs) with knowledge bases may improve medical domain-specific performance, practical methods are needed for local implementation of LLMs that address privacy concerns and enhance accessibility for health care professionals. Objective: To develop an accurate, cost-effective local implementation of an LLM to mitigate privacy concerns and support their practical deployment in health care settings. Design, setting, and participants: ChatZOC (Sun Yat-Sen University Zhongshan Ophthalmology Center), a retrieval-augmented LLM framework, was developed by enhancing a baseline LLM with a comprehensive ophthalmic dataset and evaluation framework (CODE), which includes over 30 000 pieces of ophthalmic knowledge. This LLM was benchmarked against 10 representative LLMs, including GPT-4 and GPT-3.5 Turbo (OpenAI), across 300 clinical questions in ophthalmology. The evaluation, involving a panel of medical experts and biomedical researchers, focused on accuracy, utility, and safety. A double-masked approach was used to try to minimize bias assessment across all models. The study used a comprehensive knowledge base derived from ophthalmic clinical practice, without directly involving clinical patients. Exposures: LLM response to clinical questions. Main outcomes and measures: Accuracy, utility, and safety of LLMs in responding to clinical questions. Results: The baseline model achieved a human ranking score of 0.48. The retrieval-augmented LLM had a score of 0.60, a difference of 0.12 (95% CI, 0.02-0.22; P = .02) from baseline and not different from GPT-4 with a score of 0.61 (difference = 0.01; 95% CI, -0.11 to 0.13; P = .89). For scientific consensus, the retrieval-augmented LLM was 84.0% compared with the baseline model of 46.5% (difference = 37.5%; 95% CI, 29.0%-46.0%; P < .001) and not different from GPT-4 with a value of 79.2% (difference = 4.8%; 95% CI, -0.3% to 10.0%; P = .06). Conclusions and relevance: Results of this quality improvement study suggest that the integration of high-quality knowledge bases improved the LLM's performance in medical domains. This study highlights the transformative potential of augmented LLMs in clinical practice by providing reliable, safe, and practical clinical information. Further research is needed to explore the broader application of such frameworks in the real world.

Assessment of Large Language Models in Cataract Care Information Provision: A Quantitative Comparison

Evaluating large language models in medical applications: a survey

LMOD: A Large Multimodal Ophthalmology Dataset and Benchmark for Large Vision-Language Models

Benchmarking Large Language Models for Cervical Spondylosis

Understanding natural language: Potential application of large language models to ophthalmology

Large Language Models in Healthcare: A Comprehensive Benchmark

Evaluating Large Language Models for Radiology Natural Language Processing

Development and Evaluation of a Retrieval-Augmented Large Language Model Framework for Ophthalmology

Assessing large language models' accuracy in providing patient support for choroidal melanoma

Evaluating Large Language Models in Ophthalmology

Evaluating the Performance of Different Large Language Models on Health Consultation and Patient Education in Urolithiasis

Development and evaluation of a large language model of ophthalmology in Chinese

Ophtha-LLaMA2: A Large Language Model for Ophthalmology

Benchmarking Large Language Models' Performances for Myopia Care: a Comparative Analysis of ChatGPT-3.5, ChatGPT-4.0, and Google Bard.

Large Language Model Benchmarks in Medical Tasks

Large Language Models in Ophthalmology: Potential and Pitfalls

Utilizing Large Language Models in Ophthalmology: The Current Landscape and Challenges

Can large language models be new supportive tools in coronary computed tomography angiography reporting?

Beyond the Hype: A dispassionate look at vision-language models in medical scenario

Better to Ask in English: Cross-Lingual Evaluation of Large Language Models for Healthcare Queries

ChatCAD: Interactive Computer-Aided Diagnosis on Medical Image using Large Language Models