Abstract:Accurate breast cancer prognosis prediction can help clinicians to develop appropriate treatment plans and improve life quality for patients. Recent prognostic prediction studies suggest that fusing multi-modal data, e.g., genomic data and pathological images, plays a crucial role in improving predictive performance. Despite promising results of existing approaches, there remain challenges in effective multi-modal fusion. First, albeit a powerful fusion technique, Kronecker product produces high-dimensional quadratic expansion of features that may result in high computational cost and overfitting risk, thereby limiting its performance and applicability in cancer prognosis prediction. Second, most existing methods put more attention on learning cross-modality relations between different modalities, ignoring modality-specific relations that are complementary to cross-modality relations and beneficial for cancer prognosis prediction. To address these challenges, in this study we propose a novel attention-based multi-modal network to accurately predict breast cancer prognosis, which efficiently models both modality-specific and cross-modality relations without bringing in high-dimensional features. Specifically, two intra-modality self-attentional modules and an inter-modality cross-attentional module, accompanied by latent space transformation of channel affinity matrix, are developed to successfully capture modality-specific and cross-modality relations for efficient integration of genomic data and pathological images, respectively. Moreover, we design an adaptive fusion block to take full advantage of both modality-specific and cross-modality relations. Comprehensive experiment demonstrates that our method can effectively boost prognosis prediction performance of breast cancer and compare favorably with the state-of-the-art methods.

MAIN - Multimodal Attention-based Fusion Networks for Diagnosis Prediction.

Multimodal Fusion with Cross-attention Transformer for HCC Early Recurrence Prediction from Multi-Phase CT and Clinical Data

Multimodal risk prediction with physiological signals, medical images and clinical notes

Multi-modal Fusion Network with Intra- and Inter-Modality Attention for Prognosis Prediction in Breast Cancer

Fusion of medical imaging and electronic health records with attention and multi-head machanisms

Multimodal fusion network for ICU patient outcome prediction

Automated Fusion of Multimodal Electronic Health Records for Better Medical Predictions

A feature-aware multimodal framework with auto-fusion for Alzheimer's disease diagnosis

Integrating Medical Imaging and Clinical Reports Using Multimodal Deep Learning for Advanced Disease Analysis

Deep learning and multimodal feature fusion for the aided diagnosis of Alzheimer's disease

Research on Multimodal Fusion of Temporal Electronic Medical Records

Heart failure prognosis prediction : Let's start with the MDL-HFP model

A Multimodal Affinity Fusion Network for Predicting the Survival of Breast Cancer Patients

3D Multimodal Fusion Network With Disease-Induced Joint Learning for Early Alzheimer's Disease Diagnosis

Attention-Like Multimodality Fusion With Data Augmentation for Diagnosis of Mental Disorders Using MRI

3D Multimodal Fusion Network With Disease-Induced Joint Learning for Early Alzheimer’s Disease Diagnosis

Multimodal Triplet Attention Network for Brain Disease Diagnosis

Multimodal cross enhanced fusion network for diagnosis of Alzheimer?s disease and

Multimodal Cross Enhanced Fusion Network for Diagnosis of Alzheimer's Disease and Subjective Memory Complaints.

Multimodal Fusion Learning with Dual Attention for Medical Imaging

An attention-based multi-modal MRI fusion model for major depressive disorder diagnosis