Abstract:Background and objective Due to the complexity of skin lesion features, computer-aided diagnosis of skin diseases based on multi-modal images is considered a challenging task. Dermoscopic images and clinical images are commonly used to diagnose skin diseases in clinical scenarios, and the complementarity of their features promotes the research of multi-modality classification in the computer-aided diagnosis field. Most current methods focus on the fusion between modalities and ignore the complementary information within each of them, which leads to the loss of the intra-modality relation. Multi-modality models for integrating features both within single modalities and across multiple modalities are limited in the literature. Therefore, a multi-modality model based on dermoscopic and clinical images is proposed to address this issue. Methods We propose a Multi-scale Fully-shared Fusion Network (MFF-Net) that gathers features of dermoscopic images and clinical images for skin lesion classification. In MFF-Net, the multi-scale fusion structure combines deep and shallow features within individual modalities to reduce the loss of spatial information in high-level feature maps. Then Dermo-Clinical Block (DCB) integrates the feature maps from dermoscopic images and clinical images through channel-wise concatenation and using a fully-shared fusion strategy that explores complementary information at different stages. Results We validated our model on a four-class two-modal skin diseases dataset, and proved that the proposed multi-scale structure, the fusion module DCBs, and the fully-shared fusion strategy improve the performance of MFF-Net independently. Our method achieved the highest average accuracy of 72.9% on the 7-point checklist dataset, outperforming the state-of-the-art single-modality and multi-modality methods with an accuracy boost of 7.1% and 3.4%, respectively. Conclusions The multi-scale fusion structure demonstrates the significance of intra-modality relations between clinical images and dermoscopic images. The proposed network combined with the multi-scale structure, DCBs, and the fully-shared fusion strategy, can effectively integrate the features of the skin lesions across the two modalities and achieved a promising accuracy among different skin diseases.

A novel multi-task learning network for skin lesion classification based on multi-modal clues and label-level fusion

FusionM4Net: A Multi-Stage Multi-Modal Learning Algorithm for Multi-Label Skin Lesion Classification

Joint segmentation and classification of skin lesions via a multi-task learning convolutional neural network

A Novel Multi-task Deep Learning Model for Skin Lesion Segmentation and Classification

A Deep Multi-task Learning Approach to Skin Lesion Classification

Skin Lesion classification based on two-modal images using a multi-scale fully-shared fusion network

A Novel Deep Learning Model for Breast Tumor Ultrasound Image Classification with Lesion Region Perception

MSLANet: Multi-Scale Long Attention Network for Skin Lesion Classification

MLFF-Net: a multi-model late feature fusion network for skin disease classification

A Cascade Attention Network for Liver Lesion Classification in Weakly-Labeled Multi-phase CT Images.

MSMA: A multi-stage and multi-attention algorithm for the classification of multimodal skin lesions

Fully Transformer Network for Skin Lesion Analysis

Adversarial Multimodal Fusion with Attention Mechanism for Skin Lesion Classification Using Clinical and Dermoscopic Images

A Novel Perspective for Multi-modal Multi-label Skin Lesion Classification

MASDF-Net: A Multi-Attention Codec Network with Selective and Dynamic Fusion for Skin Lesion Segmentation

MLDF-Net: Metadata Based Multi-level Dynamic Fusion Network.

A novel soft attention-based multi-modal deep learning framework for multi-label skin lesion classification

MSF-Net: A Lightweight Multi-Scale Feature Fusion Network for Skin Lesion Segmentation

TFormer: A Throughout Fusion Transformer for Multi-Modal Skin Lesion Diagnosis

3D Multi-Attention Guided Multi-Task Learning Network for Automatic Gastric Tumor Segmentation and Lymph Node Classification.

MT-TransUNet: Mediating Multi-Task Tokens in Transformers for Skin Lesion Segmentation and Classification