Abstract:Background Clinical appearance and high-frequency ultrasound (HFUS) are indispensable for diagnosing skin diseases by providing internal and external information. However, their complex combination brings challenges for primary care physicians and dermatologists. Thus, we developed a deep multimodal fusion network (DMFN) model combining analysis of clinical close-up and HFUS images for binary and multiclass classification in skin diseases. Methods Between Jan 10, 2017, and Dec 31, 2020, the DMFN model was trained and validated using 1269 close-ups and 11,852 HFUS images from 1351 skin lesions. The monomodal convolutional neural network (CNN) model was trained and validated with the same close-up images for comparison. Subsequently, we did a prospective and multicenter study in China. Both CNN models were tested prospectively on 422 cases from 4 hospitals and compared with the results from human raters (general practitioners, general dermatologists, and dermatologists specialized in HFUS). The performance of binary classification (benign vs. malignant) and multiclass classification (the specific diagnoses of 17 types of skin diseases) measured by the area under the receiver operating characteristic curve (AUC) were evaluated. This study is registered with www.chictr.org.cn (ChiCTR2300074765). Findings The performance of the DMFN model (AUC, 0.876) was superior to that of the monomodal CNN model (AUC, 0.697) in the binary classification (P = 0.0063), which was also better than that of the general practitioner (AUC, 0.651, P = 0.0025) and general dermatologists (AUC, 0.838; P = 0.0038). By integrating close-up and HFUS images, the DMFN model attained an almost identical performance in comparison to dermatologists (AUC, 0.876 vs. AUC, 0.891; P = 0.0080). For the multiclass classification, the DMFN model (AUC, 0.707) exhibited superior prediction performance compared with general dermatologists (AUC, 0.514; P = 0.0043) and dermatologists specialized in HFUS (AUC, 0.640; P = 0.0083), respectively. Compared to dermatologists specialized in HFUS, the DMFN model showed better or comparable performance in diagnosing 9 of the 17 skin diseases. Interpretation The DMFN model combining analysis of clinical close-up and HFUS images exhibited satisfactory performance in the binary and multiclass classification compared with the dermatologists. It may be a valuable tool for general dermatologists and primary care providers. Copyright (c) 2023 The Author(s). Published by Elsevier Ltd. This is an open access article under the CC BY -NC -ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).

Multi-feature Fusion Deep Network for Skin Disease Diagnosis

MLFF-Net: a multi-model late feature fusion network for skin disease classification

A Deep Learning Fusion Network Trained with Clinical and High-Frequency Ultrasound Images in the Multi-Classification of Skin Diseases in Comparison with Dermatologists: a Prospective and Multicenter Study

Skin Lesion classification based on two-modal images using a multi-scale fully-shared fusion network

A Skin Disease Classification Model Based on DenseNet and ConvNeXt Fusion

Fusion of U-Net and CNN model for segmentation and classification of skin lesion from dermoscopy images

FusionM4Net: A Multi-Stage Multi-Modal Learning Algorithm for Multi-Label Skin Lesion Classification

Deep convolutional neural network with fusion strategy for skin cancer recognition: model development and validation

Computer-Aided Diagnosis for Early Signs of Skin Diseases Using Multi Types Feature Fusion Based on a Hybrid Deep Learning Model

Multi-Model Attentional Fusion Ensemble for Accurate Skin Cancer Classification

DVFNet: A deep feature fusion-based model for the multiclassification of skin cancer utilizing dermoscopy images

A multimodal transformer to fuse images and metadata for skin disease classification

Skin Disease Classification Using Deep Learning

MobileNet-V2: An Enhanced Skin Disease Classification by Attention and Multi-Scale Features

A two‐stream deep neural network‐based intelligent system for complex skin cancer types classification

An Integrated Multimodal Deep Learning Framework for Accurate Skin Disease Classification

Skin Disease Detection Based on Deep Learning

Integration of Localized, Contextual, and Hierarchical Features in Deep Learning for Improved Skin Lesion Classification

Ensemble of weighted deep concatenated features for the skin disease classification model using modified long short term memory

Skin disease classification using two path deep transfer learning models

XAI-FusionNet: Diabetic foot ulcer detection based on multi-scale feature fusion with explainable artificial intelligence