Abstract:Background Clinical appearance and high-frequency ultrasound (HFUS) are indispensable for diagnosing skin diseases by providing internal and external information. However, their complex combination brings challenges for primary care physicians and dermatologists. Thus, we developed a deep multimodal fusion network (DMFN) model combining analysis of clinical close-up and HFUS images for binary and multiclass classification in skin diseases. Methods Between Jan 10, 2017, and Dec 31, 2020, the DMFN model was trained and validated using 1269 close-ups and 11,852 HFUS images from 1351 skin lesions. The monomodal convolutional neural network (CNN) model was trained and validated with the same close-up images for comparison. Subsequently, we did a prospective and multicenter study in China. Both CNN models were tested prospectively on 422 cases from 4 hospitals and compared with the results from human raters (general practitioners, general dermatologists, and dermatologists specialized in HFUS). The performance of binary classification (benign vs. malignant) and multiclass classification (the specific diagnoses of 17 types of skin diseases) measured by the area under the receiver operating characteristic curve (AUC) were evaluated. This study is registered with www.chictr.org.cn (ChiCTR2300074765). Findings The performance of the DMFN model (AUC, 0.876) was superior to that of the monomodal CNN model (AUC, 0.697) in the binary classification (P = 0.0063), which was also better than that of the general practitioner (AUC, 0.651, P = 0.0025) and general dermatologists (AUC, 0.838; P = 0.0038). By integrating close-up and HFUS images, the DMFN model attained an almost identical performance in comparison to dermatologists (AUC, 0.876 vs. AUC, 0.891; P = 0.0080). For the multiclass classification, the DMFN model (AUC, 0.707) exhibited superior prediction performance compared with general dermatologists (AUC, 0.514; P = 0.0043) and dermatologists specialized in HFUS (AUC, 0.640; P = 0.0083), respectively. Compared to dermatologists specialized in HFUS, the DMFN model showed better or comparable performance in diagnosing 9 of the 17 skin diseases. Interpretation The DMFN model combining analysis of clinical close-up and HFUS images exhibited satisfactory performance in the binary and multiclass classification compared with the dermatologists. It may be a valuable tool for general dermatologists and primary care providers. Copyright (c) 2023 The Author(s). Published by Elsevier Ltd. This is an open access article under the CC BY -NC -ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).

Fine-grained Lesion Classification Framework for Early Auxiliary Diagnosis

Residual Convolutional Neural Networks With Global And Local Pathways For Classification Of Focal Liver Lesions

Combining Convolutional and Recurrent Neural Networks for Classification of Focal Liver Lesions in Multi-phase CT Images.

Classification of Focal Liver Lesions Using Deep Learning with Fine-Tuning

A Deep Learning Fusion Network Trained with Clinical and High-Frequency Ultrasound Images in the Multi-Classification of Skin Diseases in Comparison with Dermatologists: a Prospective and Multicenter Study

Lesion Feature Extraction and Classification Optimization Method Using Dynamic Fusion of Global Attention and Local Attention

LACL: Lesion-Aware Contrastive Learning Framework for Medical Image Classification

A Lesion-Fusion Neural Network for Multi-View Diabetic Retinopathy Grading

A Novel Hierarchical Deep Learning Framework for Diagnosing Multiple Visual Impairment Diseases in the Clinical Environment

Attention-Guided Learning with Feature Reconstruction for Skin Lesion Diagnosis using Clinical and Ultrasound Images

Lesion Segmentation Framework Based on Convolutional Neural Networks with Dual Attention Mechanism

A Multi-Scale and Multi-Level Fusion Approach for Deep Learning-Based Liver Lesion Diagnosis in Magnetic Resonance Images with Visual Explanation

Attention to Lesion: Lesion-Aware Convolutional Neural Network for Retinal Optical Coherence Tomography Image Classification

Automated Interpretation of Fundus Fluorescein Angiography with Multi-Retinal Lesion Segmentation

Fine-Grained Classification of Colposcopic Images with Multi-Modality

Automatic Detection and Classification of Focal Liver Lesions Based on Deep Convolutional Neural Networks: A Preliminary Study

Bi-directional Dermoscopic Feature Learning and Multi-scale Consistent Decision Fusion for Skin Lesion Segmentation

Lesion-Aware Contrastive Learning for Diabetic Retinopathy Diagnosis

Automatic Analysis of Lesion in Cardiovascular Image Using Fully Convolutional Neural Networks

CAF-YOLO: A Robust Framework for Multi-Scale Lesion Detection in Biomedical Imagery

R-FCN Based Laryngeal Lesion Detection