Abstract:Multi-modal learning has shown exceptional performance in various tasks, especially in medical applications, where it integrates diverse medical information for comprehensive diagnostic evidence. However, there still are several challenges in multi-modal learning, 1. Heterogeneity between modalities, 2. uncertainty in missing modalities, 3. influence of intrinsic noise, and 4. interpretability for fusion result. This paper introduces the Modal-Domain Attention (MDA) model to address the above challenges. MDA constructs linear relationships between modalities through continuous attention, due to its ability to adaptively allocate dynamic attention to different modalities, MDA can reduce attention to low-correlation data, missing modalities, or modalities with inherent noise, thereby maintaining SOTA performance across various tasks on multiple public datasets. Furthermore, our observations on the contribution of different modalities indicate that MDA aligns with established clinical diagnostic imaging gold standards and holds promise as a reference for pathologies where these standards are not yet clearly defined. The code and dataset will be available.

What problem does this paper attempt to address?

This paper attempts to address four main challenges in multi - modal learning: 1. **Modal Heterogeneity**: The information redundancy and inconsistent complementarity between different modalities complicate the fusion process. As the number of modalities increases, the fusion process becomes more difficult. 2. **Modal Missing**: In practical applications, modal missing is an inevitable problem. Although existing methods such as generative models and prompt - based methods can handle modal missing, these methods usually require complex network structures, increasing the training difficulty. 3. **Intrinsic Noise Learning**: The intrinsic noise in multi - modal data (for example, the difference between text information and clinical reality) will introduce contradictions and affect the effect of modal fusion. Exploring how to use the complementary differences between modalities to mitigate the impact of intrinsic noise is an important issue. 4. **Comprehensive Interpretability of Multi - modal Fusion**: Current techniques such as heat maps are mainly used to improve the interpretability of single - modal models, but lack a unified interpretation paradigm in cross - modal scenarios. In medical diagnosis, a method that can comprehensively interpret the results of multi - modal fusion is required. To address the above challenges, the paper proposes the Modal - Domain Attention (MDA) model. MDA dynamically adjusts the attention weights of different modalities by constructing linear relationships between modalities, thereby reducing the attention to low - correlation data, missing modalities or modalities containing intrinsic noise, and maintaining high performance on multiple tasks. Specifically, the main contributions of MDA include: - **Adaptive Adjustment of Modal Weights**: MDA can dynamically adjust its weights according to the importance of each modality in the final decision, thereby effectively improving the performance of multi - modal tasks. - **Reducing the Influence of Low - Correlation Information**: MDA can reduce the attention to low - correlation information and minimize the influence of missing modalities and intrinsic noise on the results. - **Exploring the Interpretability of Multi - modal Learning**: Based on MDA, the research comprehensively analyzes the sources of multi - modal efficacy from two perspectives, macro (between different diseases) and micro (within individual cases), and preliminarily explores the interpretability of multi - modal learning as the gold standard for clinical diagnosis, promoting the accurate and standardized diagnosis of diseases lacking clear diagnostic criteria. Through these methods, MDA has demonstrated excellent performance on multiple public datasets, especially in handling cases of missing modalities and intrinsic noise.

MDA: An Interpretable and Scalable Multi-Modal Fusion under Missing Modalities and Intrinsic Noise Conditions

Missing-modality Enabled Multi-modal Fusion Architecture for Medical Data

M3AE: Multimodal Representation Learning for Brain Tumor Segmentation with Missing Modalities

Multimodal Fusion Learning with Dual Attention for Medical Imaging

Two-Dimensional Attentive Fusion for Multi-Modal Learning of Neuroimaging and Genomics Data

Mixture of Attention Variants for Modal Fusion in Multi-Modal Sentiment Analysis

MDC-RHT: Multi-Modal Medical Image Fusion via Multi-Dimensional Dynamic Convolution and Residual Hybrid Transformer

MSH-Net: Modality-Shared Hallucination With Joint Adaptation Distillation for Remote Sensing Image Classification Using Missing Modalities

Interpretation on Deep Multimodal Fusion for Diagnostic Classification

M$^3$Care: Learning with Missing Modalities in Multimodal Healthcare Data

DrFuse: Learning Disentangled Representation for Clinical Multi-Modal Fusion with Missing Modality and Modal Inconsistency

What Makes for Robust Multi-Modal Models in the Face of Missing Modalities?

M3Care: Learning with Missing Modalities in Multimodal Healthcare Data

Multi-modal Spatial-modality Attentive Fusion for Studying Neuropsychiatric Disorders

Cascaded Multi-Modal Mixing Transformers for Alzheimer's Disease Classification with Incomplete Data

M4FNet: Multimodal Medical Image Fusion Network Via Multi-Receptive-field and Multi-Scale Feature Integration.

MSAF: Multimodal Split Attention Fusion

Multi-modal Brain Tumor Segmentation via Missing Modality Synthesis and Modality-level Attention Fusion

Analysing and Evaluating Complementarity of Multi-Modal Data Fusion in AD Diagnosis.

Inter-modality Dependence Induced Data Recovery for MCI Conversion Prediction

3D Multimodal Fusion Network With Disease-Induced Joint Learning for Early Alzheimer's Disease Diagnosis