Abstract:Background: The information between multimodal magnetic resonance imaging (MRI) is complementary. Combining multiple modalities for brain tumor image segmentation can improve segmentation accuracy, which has great significance for disease diagnosis and treatment. However, different degrees of missing modality data often occur in clinical practice, which may lead to serious performance degradation or even failure of brain tumor segmentation methods relying on full-modality sequences to complete the segmentation task. To solve the above problems, this study aimed to design a new deep learning network for incomplete multimodal brain tumor segmentation. Methods: We propose a novel cross-modal attention fusion-based deep neural network (CMAF-Net) for incomplete multimodal brain tumor segmentation, which is based on a three-dimensional (3D) U-Net architecture with encoding and decoding structure, a 3D Swin block, and a cross-modal attention fusion (CMAF) block. A convolutional encoder is initially used to extract the specific features from different modalities, and an effective 3D Swin block is constructed to model the long-range dependencies to obtain richer information for brain tumor segmentation. Then, a cross-attention based CMAF module is proposed that can deal with different missing modality situations by fusing features between different modalities to learn the shared representations of the tumor regions. Finally, the fused latent representation is decoded to obtain the final segmentation result. Additionally, channel attention module (CAM) and spatial attention module (SAM) are incorporated into the network to further improve the robustness of the model; the CAM to help focus on important feature channels, and the SAM to learn the importance of different spatial regions. Results: Evaluation experiments on the widely-used BraTS 2018 and BraTS 2020 datasets demonstrated the effectiveness of the proposed CMAF-Net which achieved average Dice scores of 87.9%, 81.8%, and 64.3%, as well as Hausdorff distances of 4.21, 5.35, and 4.02 for whole tumor, tumor core, and enhancing tumor on the BraTS 2020 dataset, respectively, outperforming several state-of-the-art segmentation methods in missing modalities situations. Conclusions: The experimental results show that the proposed CMAF-Net can achieve accurate brain tumor segmentation in the case of missing modalities with promising application potential.

MSAIF-Net: A Multistage Spatial Attention-Based Invertible Fusion Network for MR Images.

A Multi-Stage Visible and Infrared Image Fusion Network Based on Attention Mechanism

SIMFusion: A Semantic Information-Guided Modality-Specific Fusion Network for MR Images

AMMNet: A multimodal medical image fusion method based on an attention mechanism and MobileNetV3

MBRARN: multibranch residual attention reconstruction network for medical image fusion

MAFusion: Multiscale Attention Network for Infrared and Visible Image Fusion

A layer-wise fusion network incorporating self-supervised learning for multimodal MR image synthesis

IGNFusion: An Unsupervised Information Gate Network for Multimodal Medical Image Fusion

A multibranch and multiscale neural network based on semantic perception for multimodal medical image fusion

SADFusion: A multi-scale infrared and visible image fusion method based on salient-aware and domain-specific

Adaptive spatial and frequency experts fusion network for medical image fusion

An Attention-based Multi-Scale Feature Learning Network for Multimodal Medical Image Fusion

Edge-Enhanced Dilated Residual Attention Network for Multimodal Medical Image Fusion

CMAF-Net: a cross-modal attention fusion-based deep neural network for incomplete multi-modal brain tumor segmentation

MLMFNet: A Multi-Level Modality Fusion Network for Multi-Modal Accelerated MRI Reconstruction

CMFA_Net: A cross-modal feature aggregation network for infrared-visible image fusion

Hi-Net: Hybrid-fusion Network for Multi-modal MR Image Synthesis

Semi-supervised interactive fusion network for MR image segmentation

Multi-modal medical image fusion based on densely-connected high-resolution CNN and hybrid transformer

MM-Net: A MixFormer-Based Multi-Scale Network for Anatomical and Functional Image Fusion

Multi-modal Brain Tumor Segmentation via Missing Modality Synthesis and Modality-level Attention Fusion