Abstract:Multi-modal brain tumor segmentation typically involves four magnetic resonance imaging (MRI) modalities, while incomplete modalities significantly degrade performance. Existing solutions employ explicit or implicit modality adaptation, aligning features across modalities or learning a fused feature robust to modality incompleteness. They share a common goal of encouraging each modality to express both itself and the others. However, the two expression abilities are entangled as a whole in a seamless feature space, resulting in prohibitive learning burdens. In this paper, we propose DeMoSeg to enhance the modality adaptation by Decoupling the task of representing the ego and other Modalities for robust incomplete multi-modal Segmentation. The decoupling is super lightweight by simply using two convolutions to map each modality onto four feature sub-spaces. The first sub-space expresses itself (Self-feature), while the remaining sub-spaces substitute for other modalities (Mutual-features). The Self- and Mutual-features interactively guide each other through a carefully-designed Channel-wised Sparse Self-Attention (CSSA). After that, a Radiologist-mimic Cross-modality expression Relationships (RCR) is introduced to have available modalities provide Self-feature and also `lend' their Mutual-features to compensate for the absent ones by exploiting the clinical prior knowledge. The benchmark results on BraTS2020, BraTS2018 and BraTS2015 verify the DeMoSeg's superiority thanks to the alleviated modality adaptation difficulty. Concretely, for BraTS2020, DeMoSeg increases Dice by at least 0.92%, 2.95% and 4.95% on whole tumor, tumor core and enhanced tumor regions, respectively, compared to other state-of-the-arts. Codes are at <a class="link-external link-https" href="https://github.com/kk42yy/DeMoSeg" rel="external noopener nofollow">this https URL</a>

What problem does this paper attempt to address?

The problem that this paper attempts to solve is **the performance degradation problem in multi - modal brain tumor segmentation due to incomplete modalities**. Specifically, brain tumor segmentation usually depends on four magnetic resonance imaging (MRI) modalities: T1 - weighted (T1), contrast - enhanced T1 - weighted (T1ce), T2 - weighted (T2) and fluid - attenuated inversion recovery (FLAIR). However, in actual clinical applications, factors such as image corruption, scanning protocols, and patient conditions may lead to the absence or incompleteness of some modalities, which will significantly reduce the segmentation performance of existing methods. To solve this problem, the paper proposes a new framework named **DeMoSeg**, which enhances the modality adaptation ability by decoupling the self - feature and mutual - features of each modality. The specific contributions are as follows: 1. **Feature Decoupling**: Decompose the features of each modality into self - expression features and mutual - expression features, thereby reducing the learning burden. For each modality, two convolutional layers are used to map it to four sub - spaces. The first sub - space represents self - expression features, and the remaining three sub - spaces represent the mutual - expression features of other modalities respectively. 2. **Feature Compensation based on Clinical Knowledge**: Utilize the prior knowledge of radiologists to construct pseudo - full - modality features to compensate for the influence of missing modalities. For example, when the FLAIR modality is missing, the T2 modality can provide FLAIR - specific mutual - expression features. 3. **Channel - wised Sparse Self - Attention (CSSA)**: Introduce the CSSA layer, which allows lightweight interaction between self - expression features and mutual - expression features while avoiding their re - coupling. 4. **Superior Experimental Results**: The experimental results on the BraTS2020, BraTS2018 and BraTS2015 benchmark datasets show that DeMoSeg improves the Dice coefficients in the whole tumor, tumor core and enhanced tumor regions by at least 0.92%, 2.95% and 4.95% respectively, outperforming the existing state - of - the - art methods. Through these innovations, DeMoSeg can still maintain high segmentation performance in the case of incomplete modalities, so as to better meet the challenges in clinical practice.

Decoupling Feature Representations of Ego and Other Modalities for Incomplete Multi-modal Brain Tumor Segmentation

Mmformer: Multimodal Medical Transformer for Incomplete Multimodal Learning of Brain Tumor Segmentation

M3AE: Multimodal Representation Learning for Brain Tumor Segmentation with Missing Modalities

Mutual Information-Based Graph Co-Attention Networks for Multimodal Prior-Guided Magnetic Resonance Imaging Segmentation

Robust Multimodal Brain Tumor Segmentation via Feature Disentanglement and Gated Fusion

Deformation-aware and reconstruction-driven multimodal representation learning for brain tumor segmentation with missing modalities

Multimodality-Assisted Semi-Supervised Brain Tumor Segmentation in Nondominant Modality Based on Consistency Learning

Modality-Pairing Learning for Brain Tumor Segmentation

A Multimodal Feature Distillation with CNN-Transformer Network for Brain Tumor Segmentation with Incomplete Modalities

Robust Divergence Learning for Missing-Modality Segmentation

Brain Tumor Segmentation on MRI with Missing Modalities

Multi-modal Brain Tumor Segmentation via Missing Modality Synthesis and Modality-level Attention Fusion

Enhancing Incomplete Multi-modal Brain Tumor Segmentation with Intra-modal Asymmetry and Inter-modal Dependency

Multi-modal brain tumor segmentation via disentangled representation learning and region-aware contrastive learning

Learning multi-modal brain tumor segmentation from privileged semi-paired MRI images with curriculum disentanglement learning

MedMAP: Promoting Incomplete Multi-modal Brain Tumor Segmentation with Alignment

Modality-Specific Information Disentanglement from Multi-parametric MRI for Breast Tumor Segmentation and Computer-aided Diagnosis

Unveiling Incomplete Modality Brain Tumor Segmentation: Leveraging Masked Predicted Auto-Encoder and Divergence Learning

Scratch Each Other's Back: Incomplete Multi-Modal Brain Tumor Segmentation Via Category Aware Group Self-Support Learning

Enhancing Modality-Agnostic Representations via Meta-Learning for Brain Tumor Segmentation

Exploiting Partial Common Information Microstructure for Multi-Modal Brain Tumor Segmentation