Abstract:Since the brain is the human body's primary command and control center, brain cancer is one of the most dangerous cancers. Automatic segmentation of brain tumors from multi-modal images is important in diagnosis and treatment. Due to the difficulties in obtaining multi-modal paired images in clinical practice, recent studies segment brain tumors solely relying on unpaired images and discarding the available paired images. Although these models solve the dependence on paired images, they cannot fully exploit the complementary information from different modalities, resulting in low unimodal segmentation accuracy. Hence, this work studies the unimodal segmentation with privileged semi-paired images, i.e., limited paired images are introduced to the training phase. Specifically, we present a novel two-step (intra-modality and inter-modality) curriculum disentanglement learning framework. The modality-specific style codes describe the attenuation of tissue features and image contrast, and modality-invariant content codes contain anatomical and functional information extracted from the input images. Besides, we address the problem of unthorough decoupling by introducing constraints on the style and content spaces. Experiments on the BraTS2020 dataset highlight that our model outperforms the competing models on unimodal segmentation, achieving average dice scores of 82.91%, 72.62%, and 54.80% for WT (the whole tumor), TC (the tumor core), and ET (the enhancing tumor), respectively. Finally, we further evaluate our model's variable multi-modal brain tumor segmentation performance by introducing a fusion block (TFusion). The experimental results reveal that our model achieves the best WT segmentation performance for all 15 possible modality combinations with 87.31% average accuracy. In summary, we propose a curriculum disentanglement learning framework for unimodal segmentation with privileged semi-paired images. Moreover, the benefits of the improved unimodal segmentation extend to variable multi-modal segmentation, demonstrating that improving the unimodal segmentation performance is significant for brain tumor segmentation with missing modalities. Our code is available at https://github.com/scut-cszcl/SpBTS .

Enhancing Incomplete Multi-modal Brain Tumor Segmentation with Intra-modal Asymmetry and Inter-modal Dependency

Mmformer: Multimodal Medical Transformer for Incomplete Multimodal Learning of Brain Tumor Segmentation

M3AE: Multimodal Representation Learning for Brain Tumor Segmentation with Missing Modalities

Cross-modality deep feature learning for brain tumor segmentation

Unveiling Incomplete Modality Brain Tumor Segmentation: Leveraging Masked Predicted Auto-Encoder and Divergence Learning

Learning multi-modal brain tumor segmentation from privileged semi-paired MRI images with curriculum disentanglement learning

Modality-Pairing Learning for Brain Tumor Segmentation

MedMAP: Promoting Incomplete Multi-modal Brain Tumor Segmentation with Alignment

Specific-Modal Spatial Guidance and Feature Enhancement for Multi-modal Brain Tumor Segmentation

Brain Tumor Segmentation via Multi-Modalities Interactive Feature Learning

Robust Multimodal Brain Tumor Segmentation via Feature Disentanglement and Gated Fusion

Multi-modal Brain Tumor Segmentation via Missing Modality Synthesis and Modality-level Attention Fusion

Feature-enhanced generation and multi-modality fusion based deep neural network for brain tumor segmentation with missing MR modalities

Deformation-aware and reconstruction-driven multimodal representation learning for brain tumor segmentation with missing modalities

Pre-Post Interaction Learning for Brain Tumor Segmentation with Missing MRI Modalities.

Multimodal Transformer of Incomplete MRI Data for Brain Tumor Segmentation.

Feature fusion and latent feature learning guided brain tumor segmentation and missing modality recovery network

DIGEST: Deeply supervIsed knowledGE tranSfer neTwork learning for brain tumor segmentation with incomplete multi-modal MRI scans

Brain Tumor Segmentation in Multimodal MRI Via Pixel-Level and Feature-Level Image Fusion.

multiPI-TransBTS: A Multi-Path Learning Framework for Brain Tumor Image Segmentation Based on Multi-Physical Information

A Multimodal Feature Distillation with CNN-Transformer Network for Brain Tumor Segmentation with Incomplete Modalities