Abstract:Objectives The study aimed to propose a multimodal model that incorporates both macroscopic and microscopic images and analyze its influence on clinicians’ decision-making with different levels of experience. Methods First, we constructed a multimodal dataset for five skin disorders. Next, we trained unimodal models on three different types of images and selected the best-performing models as the base learners. Then, we used a soft voting strategy to create the multimodal model. Finally, 12 clinicians were divided into three groups, with each group including one director dermatologist, one dermatologist-in-charge, one resident dermatologist, and one general practitioner. They were asked to diagnose the skin disorders in four unaided situations (macroscopic images only, dermatopathological images only, macroscopic and dermatopathological images, all images and metadata), and three aided situations (macroscopic images with model 1 aid, dermatopathological images with model 2&3 aid, all images with multimodal model 4 aid). The clinicians’ diagnosis accuracy and time for each diagnosis were recorded. Results Among the trained models, the vision transformer (ViT) achieved the best performance, with accuracies of 0.8636, 0.9545, 0.9673, and AUCs of 0.9823, 0.9952, 0.9989 on the training set, respectively. However, on the external validation set, they only achieved accuracies of 0.70, 0.90, and 0.94, respectively. The multimodal model performed well compared to the unimodal models, achieving an accuracy of 0.98 on the external validation set. The results of logit regression analysis indicate that all models are helpful to clinicians in making diagnostic decisions [Odds Ratios (OR) > 1], while metadata does not provide assistance to clinicians (OR < 1). Linear analysis results indicate that metadata significantly increases clinicians’ diagnosis time ( P < 0.05), while model assistance does not ( P > 0.05). Conclusions The results of this study suggest that the multimodal model effectively improves clinicians’ diagnostic performance without significantly increasing the diagnostic time. However, further large-scale prospective studies are necessary.

A transformer-based representation-learning model with unified processing of multimodal input for clinical diagnostics

A Transformer-based representation-learning model with unified processing of multimodal input for clinical diagnostics

Mmformer: Multimodal Medical Transformer for Incomplete Multimodal Learning of Brain Tumor Segmentation

A Transformer-Based Model for Preoperative Early Recurrence Prediction of Hepatocellular Carcinoma with Muti-modality MRI

A transformer-based unified multimodal framework for Alzheimer's disease assessment

Medical Diagnosis with Large Scale Multimodal Transformers: Leveraging Diverse Data for More Accurate Diagnosis

Learning A Multi-Task Transformer Via Unified And Customized Instruction Tuning For Chest Radiograph Interpretation

Medical transformer for multimodal survival prediction in intensive care: integration of imaging and non-imaging data

Integrating Medical Imaging and Clinical Reports Using Multimodal Deep Learning for Advanced Disease Analysis

Exploring the Influence of Transformer-Based Multimodal Modeling on Clinicians’ Diagnosis of Skin Diseases: A Quantitative Analysis

A LLM-Based Hybrid-Transformer Diagnosis System in Healthcare

COVID-19 Pneumonia Classification with Transformer from Incomplete Modalities

A multimodal transformer to fuse images and metadata for skin disease classification

Longitudinal Multimodal Transformer Integrating Imaging and Latent Clinical Signatures From Routine EHRs for Pulmonary Nodule Classification

Multimodal Diagnosis Model of Alzheimer’s Disease Based on Improved Transformer

Bidirectional Representation Learning From Transformers Using Multimodal Electronic Health Record Data to Predict Depression

Multimodal modeling with low-dose CT and clinical information for diagnostic artificial intelligence on mediastinal tumors: a preliminary study

METransformer: Radiology Report Generation by Transformer with Multiple Learnable Expert Tokens

Multi-modal Deep Learning

Medical Multi-Modal Image Transformation with Modality Code Awareness

Feature-Based Transformer with Incomplete Multimodal Brain Images for Diagnosis of Neurodegenerative Diseases