Abstract:Background Clinical appearance and high-frequency ultrasound (HFUS) are indispensable for diagnosing skin diseases by providing internal and external information. However, their complex combination brings challenges for primary care physicians and dermatologists. Thus, we developed a deep multimodal fusion network (DMFN) model combining analysis of clinical close-up and HFUS images for binary and multiclass classification in skin diseases. Methods Between Jan 10, 2017, and Dec 31, 2020, the DMFN model was trained and validated using 1269 close-ups and 11,852 HFUS images from 1351 skin lesions. The monomodal convolutional neural network (CNN) model was trained and validated with the same close-up images for comparison. Subsequently, we did a prospective and multicenter study in China. Both CNN models were tested prospectively on 422 cases from 4 hospitals and compared with the results from human raters (general practitioners, general dermatologists, and dermatologists specialized in HFUS). The performance of binary classification (benign vs. malignant) and multiclass classification (the specific diagnoses of 17 types of skin diseases) measured by the area under the receiver operating characteristic curve (AUC) were evaluated. This study is registered with www.chictr.org.cn (ChiCTR2300074765). Findings The performance of the DMFN model (AUC, 0.876) was superior to that of the monomodal CNN model (AUC, 0.697) in the binary classification (P = 0.0063), which was also better than that of the general practitioner (AUC, 0.651, P = 0.0025) and general dermatologists (AUC, 0.838; P = 0.0038). By integrating close-up and HFUS images, the DMFN model attained an almost identical performance in comparison to dermatologists (AUC, 0.876 vs. AUC, 0.891; P = 0.0080). For the multiclass classification, the DMFN model (AUC, 0.707) exhibited superior prediction performance compared with general dermatologists (AUC, 0.514; P = 0.0043) and dermatologists specialized in HFUS (AUC, 0.640; P = 0.0083), respectively. Compared to dermatologists specialized in HFUS, the DMFN model showed better or comparable performance in diagnosing 9 of the 17 skin diseases. Interpretation The DMFN model combining analysis of clinical close-up and HFUS images exhibited satisfactory performance in the binary and multiclass classification compared with the dermatologists. It may be a valuable tool for general dermatologists and primary care providers. Copyright (c) 2023 The Author(s). Published by Elsevier Ltd. This is an open access article under the CC BY -NC -ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).

MF-OMKT: Model Fusion Based on Online Mutual Knowledge Transfer for Breast Cancer Histopathological Image Classification.

MDFF-Net: A multi-dimensional feature fusion network for breast histopathology image classification

Richer fusion network for breast cancer classification based on multimodal data

Mammographic image classification with deep fusion learning

MBFusion: Multi-modal balanced fusion and multi-task learning for cancer diagnosis and prognosis

Constructing the Optimal Classification Model for Benign and Malignant Breast Tumors Based on Multifeature Analysis from Multimodal Images

Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosis

M<SUP>2</SUP>F: A Multi-modal and Multi-task Fusion Network for Glioma Diagnosis and Prognosis

MM-DCNet: A Multi-Scale and Multi-MOdality Dynamic Convolutional Network for Lung Cancer Subtypes Classification

A Deep Learning Fusion Network Trained with Clinical and High-Frequency Ultrasound Images in the Multi-Classification of Skin Diseases in Comparison with Dermatologists: a Prospective and Multicenter Study

Integration of Multimodal Data for Breast Cancer Classification Using a Hybrid Deep Learning Method

Hierarchical multimodal fusion framework based on noisy label learning and attention mechanism for cancer classification with pathology and genomic features

Pathological Image Classification Via Embedded Fusion Mutual Learning

Multi-path synergic fusion deep neural network framework for breast mass classification using digital breast tomosynthesis

MSMFN: An Ultrasound Based Multi-Step Modality Fusion Network for Identifying the Histologic Subtypes of Metastatic Cervical Lymphadenopathy

Improving HER2-Positive Breast Cancer Targeted Therapy Prediction Using Multimsnet: A Multi-Scale Pathological Image-Based Approach

Multi-modal Multi-instance Learning Using Weakly Correlated Histopathological Images and Tabular Clinical Information

OMG-Net: A Deep Learning Framework Deploying Segment Anything to Detect Pan-Cancer Mitotic Figures from Haematoxylin and Eosin-Stained Slides

PMFN-SSL: Self-supervised learning-based progressive multimodal fusion network for cancer diagnosis and prognosis

Multimodal microscopic imaging with deep learning for highly effective diagnosis of breast cancer

Federated Fusion of Magnified Histopathological Images for Breast Tumor Classification in the Internet of Medical Things