Abstract:Background Clinical appearance and high-frequency ultrasound (HFUS) are indispensable for diagnosing skin diseases by providing internal and external information. However, their complex combination brings challenges for primary care physicians and dermatologists. Thus, we developed a deep multimodal fusion network (DMFN) model combining analysis of clinical close-up and HFUS images for binary and multiclass classification in skin diseases. Methods Between Jan 10, 2017, and Dec 31, 2020, the DMFN model was trained and validated using 1269 close-ups and 11,852 HFUS images from 1351 skin lesions. The monomodal convolutional neural network (CNN) model was trained and validated with the same close-up images for comparison. Subsequently, we did a prospective and multicenter study in China. Both CNN models were tested prospectively on 422 cases from 4 hospitals and compared with the results from human raters (general practitioners, general dermatologists, and dermatologists specialized in HFUS). The performance of binary classification (benign vs. malignant) and multiclass classification (the specific diagnoses of 17 types of skin diseases) measured by the area under the receiver operating characteristic curve (AUC) were evaluated. This study is registered with www.chictr.org.cn (ChiCTR2300074765). Findings The performance of the DMFN model (AUC, 0.876) was superior to that of the monomodal CNN model (AUC, 0.697) in the binary classification (P = 0.0063), which was also better than that of the general practitioner (AUC, 0.651, P = 0.0025) and general dermatologists (AUC, 0.838; P = 0.0038). By integrating close-up and HFUS images, the DMFN model attained an almost identical performance in comparison to dermatologists (AUC, 0.876 vs. AUC, 0.891; P = 0.0080). For the multiclass classification, the DMFN model (AUC, 0.707) exhibited superior prediction performance compared with general dermatologists (AUC, 0.514; P = 0.0043) and dermatologists specialized in HFUS (AUC, 0.640; P = 0.0083), respectively. Compared to dermatologists specialized in HFUS, the DMFN model showed better or comparable performance in diagnosing 9 of the 17 skin diseases. Interpretation The DMFN model combining analysis of clinical close-up and HFUS images exhibited satisfactory performance in the binary and multiclass classification compared with the dermatologists. It may be a valuable tool for general dermatologists and primary care providers. Copyright (c) 2023 The Author(s). Published by Elsevier Ltd. This is an open access article under the CC BY -NC -ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).

Application of Multimodal Fusion Deep Learning Model in Disease Recognition

Integrating Medical Imaging and Clinical Reports Using Multimodal Deep Learning for Advanced Disease Analysis

Interpretation on Deep Multimodal Fusion for Diagnostic Classification

Enhancing Heart Failure Diagnosis Through Multi-Modal Data Integration and Deep Learning

Deep Multi-modal Fusion of Image and Non-image Data in Disease Diagnosis and Prognosis: A Review

Disease Classification Model Based on Multi-Modal Feature Fusion

Multimodal Medical Image Fusion: The Perspective of Deep Learning

A Deep Learning Fusion Network Trained with Clinical and High-Frequency Ultrasound Images in the Multi-Classification of Skin Diseases in Comparison with Dermatologists: a Prospective and Multicenter Study

Multimodal medical image fusion and classification using deep learning techniques

Integration of Multimodal Data for Breast Cancer Classification Using a Hybrid Deep Learning Method

Multimodal cross enhanced fusion network for diagnosis of Alzheimer?s disease and

Multimodal Cross Enhanced Fusion Network for Diagnosis of Alzheimer's Disease and Subjective Memory Complaints.

Multimodal medical image fusion using convolutional neural network and extreme learning machine

Multimodal fusion with deep neural networks for leveraging CT imaging and electronic health record: a case-study in pulmonary embolism detection

A review of deep learning-based information fusion techniques for multimodal medical image classification

Deep learning and multimodal feature fusion for the aided diagnosis of Alzheimer's disease

Multimodal Fusion Learning with Dual Attention for Medical Imaging

Missing-modality Enabled Multi-modal Fusion Architecture for Medical Data

Deep Learning Based Multimodal Biomedical Data Fusion: an Overview and Comparative Review

Transformer-Based Multi-Modal Data Fusion Method for COPD Classification and Physiological and Biochemical Indicators Identification

Multi-Modal Multi-Instance Learning for Retinal Disease Recognition