Abstract:There is a wide application of deep learning technique to unimodal medical image analysis with significant classification accuracy performance observed. However, real-world diagnosis of some chronic diseases such as breast cancer often require multimodal data streams with different modalities of visual and textual content. Mammography, magnetic resonance imaging (MRI) and image-guided breast biopsy represent a few of multimodal visual streams considered by physicians in isolating cases of breast cancer. Unfortunately, most studies applying deep learning techniques to solving classification problems in digital breast images have often narrowed their study to unimodal samples. This is understood considering the challenging nature of multimodal image abnormality classification where the fusion of high dimension heterogeneous features learned needs to be projected into a common representation space. This paper presents a novel deep learning approach combining a dual/twin convolutional neural network (TwinCNN) framework to address the challenge of breast cancer image classification from multi-modalities. First, modality-based feature learning was achieved by extracting both low and high levels features using the networks embedded with TwinCNN. Secondly, to address the notorious problem of high dimensionality associated with the extracted features, binary optimization method is adapted to effectively eliminate non-discriminant features in the search space. Furthermore, a novel method for feature fusion is applied to computationally leverage the ground-truth and predicted labels for each sample to enable multimodality classification. To evaluate the proposed method, digital mammography images and digital histopathology breast biopsy samples from benchmark datasets namely MIAS and BreakHis respectively. Experimental results obtained showed that the classification accuracy and area under the curve (AUC) for the single modalities yielded 0.755 and 0.861871 for histology, and 0.791 and 0.638 for mammography. Furthermore, the study investigated classification accuracy resulting from the fused feature method, and the result obtained showed that 0.977, 0.913, and 0.667 for histology, mammography, and multimodality respectively. The findings from the study confirmed that multimodal image classification based on combination of image features and predicted label improves performance. In addition, the contribution of the study shows that feature dimensionality reduction based on binary optimizer supports the elimination of non-discriminant features capable of bottle-necking the classifier.

A comprehensive investigation of multimodal deep learning fusion strategies for breast cancer classification

Integration of Multimodal Data for Breast Cancer Classification Using a Hybrid Deep Learning Method

A twin convolutional neural network with hybrid binary optimizer for multimodal breast cancer digital image classification

Deep Learning Based Multimodal Biomedical Data Fusion: an Overview and Comparative Review

Multimodal deep learning for biomedical data fusion: a review

A review of deep learning-based information fusion techniques for multimodal medical image classification

A deep feature fusion methodology for breast cancer diagnosis demonstrated on three imaging modality datasets

Richer fusion network for breast cancer classification based on multimodal data

An overview of deep learning methods for multimodal medical data mining

Deep Multi-modal Fusion of Image and Non-image Data in Disease Diagnosis and Prognosis: A Review

Deep learning approaches to detect breast cancer: a comprehensive review

Multi-path synergic fusion deep neural network framework for breast mass classification using digital breast tomosynthesis

Histopathology in focus: a review on explainable multi-modal approaches for breast cancer diagnosis

Multimodal Machine Learning in Precision Health

Deep learning‐based multimodal fusion network for segmentation and classification of breast cancers using B‐mode and elastography ultrasound images

Breast Cancer Classification from Ultrasound Images Using Probability-Based Optimal Deep Learning Feature Fusion

A Brief Survey On Breast Cancer Diagnostic With Deep Learning Schemes Using Multi-Image Modalities

Diagnosis of breast cancer molecular subtypes using machine learning models on unimodal and multimodal datasets

Multimodal classification of breast cancer using feature level fusion of mammogram and ultrasound images in machine learning paradigm

Enhanced breast cancer diagnosis through integration of computer vision with fusion based joint transfer learning using multi modality medical images

Multimodal Medical Image Fusion: The Perspective of Deep Learning