A Multimodal Feature Distillation with CNN-Transformer Network for Brain Tumor Segmentation with Incomplete Modalities

Ming Kang,Fung Fung Ting,Raphaël C.-W. Phan,Zongyuan Ge,Chee-Ming Ting

2024-04-22

Abstract:Existing brain tumor segmentation methods usually utilize multiple Magnetic Resonance Imaging (MRI) modalities in brain tumor images for segmentation, which can achieve better segmentation performance. However, in clinical applications, some modalities are missing due to resource constraints, leading to severe degradation in the performance of methods applying complete modality segmentation. In this paper, we propose a Multimodal feature distillation with Convolutional Neural Network (CNN)-Transformer hybrid network (MCTSeg) for accurate brain tumor segmentation with missing modalities. We first design a Multimodal Feature Distillation (MFD) module to distill feature-level multimodal knowledge into different unimodality to extract complete modality information. We further develop a Unimodal Feature Enhancement (UFE) module to model the relationship between global and local information semantically. Finally, we build a Cross-Modal Fusion (CMF) module to explicitly align the global correlations among different modalities even when some modalities are missing. Complementary features within and across different modalities are refined via the CNN-Transformer hybrid architectures in both the UFE and CMF modules, where local and global dependencies are both captured. Our ablation study demonstrates the importance of the proposed modules with CNN-Transformer networks and the convolutional blocks in Transformer for improving the performance of brain tumor segmentation with missing modalities. Extensive experiments on the BraTS2018 and BraTS2020 datasets show that the proposed MCTSeg framework outperforms the state-of-the-art methods in missing modalities cases. Our code is available at:

Computer Vision and Pattern Recognition,Signal Processing,Applications

What problem does this paper attempt to address?

This paper aims to address the issues encountered in brain tumor segmentation, especially the performance degradation caused by incomplete multi-modal MRI (Magnetic Resonance Imaging) data. Existing brain tumor segmentation methods typically rely on multiple MRI modalities to improve segmentation accuracy. However, in clinical applications, the lack of certain modalities due to resource limitations can severely affect the performance of these methods.

To tackle this problem, the paper proposes a new framework called MCTSeg, which combines Convolutional Neural Networks (CNN) and Transformer architecture. MCTSeg consists of three innovative modules: 1) Multi-modal Feature Distillation (MFD) module, which extracts knowledge from multi-modal data and transfers it to single-modal encoders; 2) Uni-modal Feature Enhancement (UFE) module, which utilizes Transformers to capture semantic relationships between global and local information; 3) Cross-modal Fusion (CMF) module, which explicitly aligns the global correlation among different modalities even when some modalities are missing.

With these modules, MCTSeg refines complementary features between different modalities in a CNN-Transformer hybrid architecture and captures both local and global dependencies. Experimental results show that MCTSeg outperforms existing state-of-the-art methods on the BraTS2018 and BraTS2020 datasets, even in the case of missing modalities.

In summary, the objective of this paper is to develop a method for accurate brain tumor segmentation even in the presence of incomplete MRI data, addressing challenges in clinical applications.