Abstract:There is a wide application of deep learning technique to unimodal medical image analysis with significant classification accuracy performance observed. However, real-world diagnosis of some chronic diseases such as breast cancer often require multimodal data streams with different modalities of visual and textual content. Mammography, magnetic resonance imaging (MRI) and image-guided breast biopsy represent a few of multimodal visual streams considered by physicians in isolating cases of breast cancer. Unfortunately, most studies applying deep learning techniques to solving classification problems in digital breast images have often narrowed their study to unimodal samples. This is understood considering the challenging nature of multimodal image abnormality classification where the fusion of high dimension heterogeneous features learned needs to be projected into a common representation space. This paper presents a novel deep learning approach combining a dual/twin convolutional neural network (TwinCNN) framework to address the challenge of breast cancer image classification from multi-modalities. First, modality-based feature learning was achieved by extracting both low and high levels features using the networks embedded with TwinCNN. Secondly, to address the notorious problem of high dimensionality associated with the extracted features, binary optimization method is adapted to effectively eliminate non-discriminant features in the search space. Furthermore, a novel method for feature fusion is applied to computationally leverage the ground-truth and predicted labels for each sample to enable multimodality classification. To evaluate the proposed method, digital mammography images and digital histopathology breast biopsy samples from benchmark datasets namely MIAS and BreakHis respectively. Experimental results obtained showed that the classification accuracy and area under the curve (AUC) for the single modalities yielded 0.755 and 0.861871 for histology, and 0.791 and 0.638 for mammography. Furthermore, the study investigated classification accuracy resulting from the fused feature method, and the result obtained showed that 0.977, 0.913, and 0.667 for histology, mammography, and multimodality respectively. The findings from the study confirmed that multimodal image classification based on combination of image features and predicted label improves performance. In addition, the contribution of the study shows that feature dimensionality reduction based on binary optimizer supports the elimination of non-discriminant features capable of bottle-necking the classifier.

Integration of Multimodal Data for Breast Cancer Classification Using a Hybrid Deep Learning Method

Richer fusion network for breast cancer classification based on multimodal data

Multimodal microscopic imaging with deep learning for highly effective diagnosis of breast cancer

A comprehensive investigation of multimodal deep learning fusion strategies for breast cancer classification

Breast Cancer Histopathological Image Classification Using a Hybrid Deep Neural Network.

A Hybrid Convolutional and Recurrent Deep Neural Network for Breast Cancer Pathological Image Classification

Hierarchical multimodal fusion framework based on noisy label learning and attention mechanism for cancer classification with pathology and genomic features

A twin convolutional neural network with hybrid binary optimizer for multimodal breast cancer digital image classification

Breast Cancer Multi-classification from Histopathological Images with Structured Deep Learning Model

Deep Learning-Based Multimodal Image Analysis for Cervical Cancer Detection.

Integrating Medical Imaging and Clinical Reports Using Multimodal Deep Learning for Advanced Disease Analysis

Survey on deep learning in multimodal medical imaging for cancer detection

Deep Learning-Based Multi-Class Classification of Breast Digital Pathology Images

A Hybrid Deep Learning Model for Breast Cancer Detection and Classification

A Hybrid Deep Learning Model for Predicting Molecular Subtypes of Human Breast Cancer Using Multimodal Data

Integration of Multimodal Data from Disparate Sources for Identifying Disease Subtypes

Hybrid ensemble deep learning model for advancing breast cancer detection and classification in clinical applications

Study on the differential diagnosis of benign and malignant breast lesions using a deep learning model based on multimodal images

Deep learning‐based multimodal fusion network for segmentation and classification of breast cancers using B‐mode and elastography ultrasound images

Application of Multimodal Fusion Deep Learning Model in Disease Recognition

Improving HER2-Positive Breast Cancer Targeted Therapy Prediction Using Multimsnet: A Multi-Scale Pathological Image-Based Approach