Abstract:Background: Deep learning methods have shown great potential in processing multi-modal Magnetic Resonance Imaging (MRI) data, enabling improved accuracy in brain tumor segmentation. However, the performance of these methods can suffer when dealing with incomplete modalities, which is a common issue in clinical practice. Existing solutions, such as missing modality synthesis, knowledge distillation, and architecture-based methods, suffer from drawbacks such as long training times, high model complexity, and poor scalability. Method: This paper proposes IMS 2 Trans, a novel lightweight scalable Swin Transformer network by utilizing a single encoder to extract latent feature maps from all available modalities. This unified feature extraction process enables efficient information sharing and fusion among the modalities, resulting in efficiency without compromising segmentation performance even in the presence of missing modalities. Results: Two datasets, BraTS 2018 and BraTS 2020, containing incomplete modalities for brain tumor segmentation are evaluated against popular benchmarks. On the BraTS 2018 dataset, our model achieved higher average Dice similarity coefficient (DSC) scores for the whole tumor, tumor core, and enhancing tumor regions (86.57, 75.67, and 58.28, respectively), in comparison with a state-of-the-art model, i.e. mmFormer (86.45, 75.51, and 57.79, respectively). Similarly, on the BraTS 2020 dataset, our model scored higher DSC scores in these three brain tumor regions (87.33, 79.09, and 62.11, respectively) compared to mmFormer (86.17, 78.34, and 60.36, respectively). We also conducted a Wilcoxon test on the experimental results, and the generated p -value confirmed that our model's performance was statistically significant. Moreover, our model exhibits significantly reduced complexity with only 4.47 M parameters, 121.89G FLOPs, and a model size of 77.13 MB, whereas mmFormer comprises 34.96 M parameters, 265.79 G FLOPs, and a model size of 559.74 MB. These indicate our model, being light-weighted with significantly reduced parameters, is still able to achieve better performance than a state-of-the-art model. Conclusion: By leveraging a single encoder for processing the available modalities, IMS 2 Trans offers notable scalability advantages over methods that rely on multiple encoders. This streamlined approach eliminates the need for maintaining separate encoders for each modality, resulting in a lightweight and scalable network architecture. The source code of IMS 2 Trans and the associated weights are both publicly available at https://github.com/hudscomdz/IMS2Trans .

Multimodal Transformer of Incomplete MRI Data for Brain Tumor Segmentation.

Mmformer: Multimodal Medical Transformer for Incomplete Multimodal Learning of Brain Tumor Segmentation

MMMViT: Multiscale multimodal vision transformer for brain tumor segmentation with missing modalities

A Multimodal Feature Distillation with CNN-Transformer Network for Brain Tumor Segmentation with Incomplete Modalities

M3AE: Multimodal Representation Learning for Brain Tumor Segmentation with Missing Modalities

Mixture-of-experts and semantic-guided network for brain tumor segmentation with missing MRI modalities

Augmented Transformer network for MRI brain tumor segmentation

Deformation-aware and reconstruction-driven multimodal representation learning for brain tumor segmentation with missing modalities

Scalable Swin Transformer network for brain tumor segmentation from incomplete MRI modalities

TransBTS: Multimodal Brain Tumor Segmentation Using Transformer

CMAF-Net: a cross-modal attention fusion-based deep neural network for incomplete multi-modal brain tumor segmentation

TransSea: Hybrid CNN-Transformer with Semantic Awareness for 3D Brain Tumor Segmentation

TransSea: Hybrid CNN–Transformer With Semantic Awareness for 3-D Brain Tumor Segmentation

DIGEST: Deeply supervIsed knowledGE tranSfer neTwork learning for brain tumor segmentation with incomplete multi-modal MRI scans

Brain tumor segmentation by combining MultiEncoder UNet with wavelet fusion

Feature-Based Transformer with Incomplete Multimodal Brain Images for Diagnosis of Neurodegenerative Diseases

MM-UNet: A multimodality brain tumor segmentation network in MRI images

Brain Tumor Segmentation on MRI with Missing Modalities

3D Brainformer: 3D Fusion Transformer for Brain Tumor Segmentation

Enhancing Incomplete Multi-modal Brain Tumor Segmentation with Intra-modal Asymmetry and Inter-modal Dependency

MM-BiFPN: Multi-Modality Fusion Network With Bi-FPN for MRI Brain Tumor Segmentation