Abstract:Background: Deep learning methods have shown great potential in processing multi-modal Magnetic Resonance Imaging (MRI) data, enabling improved accuracy in brain tumor segmentation. However, the performance of these methods can suffer when dealing with incomplete modalities, which is a common issue in clinical practice. Existing solutions, such as missing modality synthesis, knowledge distillation, and architecture-based methods, suffer from drawbacks such as long training times, high model complexity, and poor scalability. Method: This paper proposes IMS 2 Trans, a novel lightweight scalable Swin Transformer network by utilizing a single encoder to extract latent feature maps from all available modalities. This unified feature extraction process enables efficient information sharing and fusion among the modalities, resulting in efficiency without compromising segmentation performance even in the presence of missing modalities. Results: Two datasets, BraTS 2018 and BraTS 2020, containing incomplete modalities for brain tumor segmentation are evaluated against popular benchmarks. On the BraTS 2018 dataset, our model achieved higher average Dice similarity coefficient (DSC) scores for the whole tumor, tumor core, and enhancing tumor regions (86.57, 75.67, and 58.28, respectively), in comparison with a state-of-the-art model, i.e. mmFormer (86.45, 75.51, and 57.79, respectively). Similarly, on the BraTS 2020 dataset, our model scored higher DSC scores in these three brain tumor regions (87.33, 79.09, and 62.11, respectively) compared to mmFormer (86.17, 78.34, and 60.36, respectively). We also conducted a Wilcoxon test on the experimental results, and the generated p -value confirmed that our model's performance was statistically significant. Moreover, our model exhibits significantly reduced complexity with only 4.47 M parameters, 121.89G FLOPs, and a model size of 77.13 MB, whereas mmFormer comprises 34.96 M parameters, 265.79 G FLOPs, and a model size of 559.74 MB. These indicate our model, being light-weighted with significantly reduced parameters, is still able to achieve better performance than a state-of-the-art model. Conclusion: By leveraging a single encoder for processing the available modalities, IMS 2 Trans offers notable scalability advantages over methods that rely on multiple encoders. This streamlined approach eliminates the need for maintaining separate encoders for each modality, resulting in a lightweight and scalable network architecture. The source code of IMS 2 Trans and the associated weights are both publicly available at https://github.com/hudscomdz/IMS2Trans .

Mmformer: Multimodal Medical Transformer for Incomplete Multimodal Learning of Brain Tumor Segmentation

Multimodal Transformer of Incomplete MRI Data for Brain Tumor Segmentation.

MMMViT: Multiscale multimodal vision transformer for brain tumor segmentation with missing modalities

M3AE: Multimodal Representation Learning for Brain Tumor Segmentation with Missing Modalities

TMFormer: Token Merging Transformer for Brain Tumor Segmentation with Missing Modalities

M2FTrans: Modality-Masked Fusion Transformer for Incomplete Multi-Modality BrainT Umor Segmentation

A Multimodal Feature Distillation with CNN-Transformer Network for Brain Tumor Segmentation with Incomplete Modalities

NestedFormer: Nested Modality-Aware Transformer for Brain Tumor Segmentation

M $^{2}$ FTrans: Modality-Masked Fusion Transformer for Incomplete Multi-Modality Brain Tumor Segmentation

3D Brainformer: 3D Fusion Transformer for Brain Tumor Segmentation

Query Re-Training for Modality-Gnostic Incomplete Multi-modal Brain Tumor Segmentation.

Scalable Swin Transformer network for brain tumor segmentation from incomplete MRI modalities

Deformation-aware and reconstruction-driven multimodal representation learning for brain tumor segmentation with missing modalities

TransSea: Hybrid CNN-Transformer with Semantic Awareness for 3D Brain Tumor Segmentation

MdcFormer: Transformers Based on Dynamic Weights and Multi-Scale for Medical Image Segmentation

TransSea: Hybrid CNN–Transformer With Semantic Awareness for 3-D Brain Tumor Segmentation

Mixture-of-experts and semantic-guided network for brain tumor segmentation with missing MRI modalities

MM-BiFPN: Multi-Modality Fusion Network With Bi-FPN for MRI Brain Tumor Segmentation

Generative Learning-Based Lightweight MRI Brain Tumor Segmentation with Missing Modalities

TransBTS: Multimodal Brain Tumor Segmentation Using Transformer

Multi-modal Brain Tumor Segmentation via Missing Modality Synthesis and Modality-level Attention Fusion