Abstract:Background: The information between multimodal magnetic resonance imaging (MRI) is complementary. Combining multiple modalities for brain tumor image segmentation can improve segmentation accuracy, which has great significance for disease diagnosis and treatment. However, different degrees of missing modality data often occur in clinical practice, which may lead to serious performance degradation or even failure of brain tumor segmentation methods relying on full-modality sequences to complete the segmentation task. To solve the above problems, this study aimed to design a new deep learning network for incomplete multimodal brain tumor segmentation. Methods: We propose a novel cross-modal attention fusion-based deep neural network (CMAF-Net) for incomplete multimodal brain tumor segmentation, which is based on a three-dimensional (3D) U-Net architecture with encoding and decoding structure, a 3D Swin block, and a cross-modal attention fusion (CMAF) block. A convolutional encoder is initially used to extract the specific features from different modalities, and an effective 3D Swin block is constructed to model the long-range dependencies to obtain richer information for brain tumor segmentation. Then, a cross-attention based CMAF module is proposed that can deal with different missing modality situations by fusing features between different modalities to learn the shared representations of the tumor regions. Finally, the fused latent representation is decoded to obtain the final segmentation result. Additionally, channel attention module (CAM) and spatial attention module (SAM) are incorporated into the network to further improve the robustness of the model; the CAM to help focus on important feature channels, and the SAM to learn the importance of different spatial regions. Results: Evaluation experiments on the widely-used BraTS 2018 and BraTS 2020 datasets demonstrated the effectiveness of the proposed CMAF-Net which achieved average Dice scores of 87.9%, 81.8%, and 64.3%, as well as Hausdorff distances of 4.21, 5.35, and 4.02 for whole tumor, tumor core, and enhancing tumor on the BraTS 2020 dataset, respectively, outperforming several state-of-the-art segmentation methods in missing modalities situations. Conclusions: The experimental results show that the proposed CMAF-Net can achieve accurate brain tumor segmentation in the case of missing modalities with promising application potential.

RFTNet: Region–Attention Fusion Network Combined with Dual-Branch Vision Transformer for Multimodal Brain Tumor Image Segmentation

RFNet: Region-aware Fusion Network for Incomplete Multi-modal Brain Tumor Segmentation

RMTF-Net: Residual Mix Transformer Fusion Net for 2D Brain Tumor Segmentation

DPAFNet: A Residual Dual-Path Attention-Fusion Convolutional Neural Network for Multimodal Brain Tumor Segmentation

OdCFU-Net for Brain Tumor Segmentation Based on MRI

LGMSU-Net: Local Features, Global Features, and Multi-Scale Features Fused the U-Shaped Network for Brain Tumor Segmentation

Modality-level cross-connection and attentional feature fusion based deep neural network for multi-modal brain tumor segmentation

Multi -Modality Brain Tumor Segmentation Network Based on Collaborative Feature Fusion

SARFNet: Selective Layer and Axial Receptive Field Network for Multimodal Brain Tumor Segmentation

RDAU-Net: Based on a Residual Convolutional Neural Network With DFP and CBAM for Brain Tumor Segmentation

Brain Tumor Segmentation in Multimodal MRI Via Pixel-Level and Feature-Level Image Fusion.

3D Brainformer: 3D Fusion Transformer for Brain Tumor Segmentation

Deep Convolutional Neural Network with a Multi-Scale Attention Feature Fusion Module for Segmentation of Multimodal Brain Tumor.

DFP-ResUNet:Convolutional Neural Network with a Dilated Convolutional Feature Pyramid for Multimodal Brain Tumor Segmentation

A multi-modality fusion network based on attention mechanism for brain tumor segmentation

MSFR-Net: Multi-modality and single-modality feature recalibration network for brain tumor segmentation

FRNET: an Effective Hybrid Structure for Automatic Segmentation of Head and Neck Primary Tumors from Multimodal Images.

MFD-Net: Modality Fusion Diffractive Network for Segmentation of Multimodal Brain Tumor Image

CMAF-Net: a cross-modal attention fusion-based deep neural network for incomplete multi-modal brain tumor segmentation

Brain Tumor Segmentation Using a Fully Convolutional Neural Network with Conditional Random Fields.

CKD-TransBTS: Clinical Knowledge-Driven Hybrid Transformer with Modality-Correlated Cross-Attention for Brain Tumor Segmentation