Abstract:Background: The introduction of ChatGPT-4V's 'Chat with images' feature represents the beginning of the era of large multimodal models (LMMs), which allows ChatGPT to process and answer questions based on uploaded images. This advancement has the potential to transform how surgical teams utilize radiographic data, as radiological interpretation is crucial for surgical planning and postoperative care. However, a comprehensive evaluation of ChatGPT-4V's capabilities in interpret radiological images and formulating treatment plans remains to be explored. Patients and methods: Three types of questions were collected: (1) 87 USMLE-style questions, submitting only the question stems and images without providing options to assess ChatGPT's diagnostic capability. For questions involving treatment plan formulations, a five-point Likert scale was used to assess ChatGPT's proposed treatment plan. The 87 questions were then adapted by removing detailed patient history to assess its contribution to diagnosis. The diagnostic performance of ChatGPT-4V was also tested when only medical history was provided. (2) We randomly selected 100 chest radiography from the ChestX-ray8 database to test the ability of ChatGPT-4V to identify abnormal chest radiography. (3) Cases from the 'Diagnose Please' section in the Radiology journal were collected to evaluate the performance of ChatGPT-4V in diagnosing complex cases. Three responses were collected for each question. Results: ChatGPT-4V achieved a diagnostic accuracy of 77.01% for USMLE-style questions. The average score of ChatGPT-4V's treatment plans was 3.97 (Interquartile Range: 3.33-4.67). Removing detailed patient history dropped the diagnostic accuracy to 19.54% (P<0.0001). ChatGPT-4V achieved an AUC of 0.768 (95% CI: 0.684-0.851) in detecting abnormalities in chest radiography, but could not specify the exact disease due to the lack of detailed patient history. For cases from 'Diagnose Please' ChatGPT provided diagnoses consistent with or very similar to the reference answers. Conclusion: ChatGPT-4V demonstrated an impressive ability to combine patient history with radiological images to make diagnoses and directly design treatment plans based on images, suggesting its potential for future application in clinical practice.

Assessing the feasibility of ChatGPT-4o and Claude 3-Opus in thyroid nodule classification based on ultrasound images

[Exploration on an Ultrasonographic Imaging Reporting and Data System in Malignancy Grading of Thyroid Nodules].

Clinical application potential of large language model: a study based on thyroid nodules

Collaborative Enhancement of Consistency and Accuracy in US Diagnosis of Thyroid Nodules Using Large Language Models

The Feasibility Of Classification Of Thyroid Nodules Integrated Expriences Based Inference Of Radiologist And Extracted Feature Vectors In Ultrasound Images

Transforming free-text radiology reports into structured reports using ChatGPT: A study on thyroid ultrasonography

Evaluation of large language models in breast cancer clinical scenarios: A comparative analysis based on ChatGPT-3.5, ChatGPT-4.0, and Claude2

Evaluating AI Proficiency in Nuclear Cardiology: Large Language Models take on the Board Preparation Exam

AI-Generated Content Enhanced Computer-Aided Diagnosis Model for Thyroid Nodules: A ChatGPT-Style Assistant

Large language model (ChatGPT) as a support tool for breast tumor board

Step into the era of large multimodal models: a pilot study on ChatGPT-4V(ision)'s ability to interpret radiological images

Empowering Radiologists with ChatGPT-4o: Comparative Evaluation of Large Language Models and Radiologists in Cardiac Cases

Extracting Thyroid Nodules Characteristics from Ultrasound Reports Using Transformer-based Natural Language Processing Methods

Ultrasound Image Classification of Thyroid Nodules Based on Deep Learning

Preliminary experiments on interpretable ChatGPT-assisted diagnosis for breast ultrasound radiologists

Deep learning diagnostic performance and visual insights in differentiating benign and malignant thyroid nodules on ultrasound images

Assessing the use of the novel tool Claude 3 in comparison to ChatGPT 4.0 as an artificial intelligence tool in the diagnosis and therapy of primary head and neck cancer cases

Non-invasive automated 3D thyroid lesion classification in ultrasound: a class of ThyroScan™ systems

Evaluation of ChatGPT Family of Models for Biomedical Reasoning and Classification

Video-based AI module with raw-scale and ROI-scale information for thyroid nodule diagnosis

Development of a privacy preserving large language model for automated data extraction from thyroid cancer pathology reports