Abstract:Glioma is a kind of brain disease with high incidence, high recurrence rate, high mortality, and low cure rate. To obtain accurate diagnosis results of brain glioma, doctors need to manually compare the imaging results of different modalities many times, which will increase the diagnosis time and reduce the diagnostic efficiency. Image fusion technology has been widely used in recent years to obtain information on multimodal medical images. This paper proposes a novel image fusion framework, target information enhanced image fusion network (TIEF), using cross-modal learning and information enhancement techniques. The framework consists of a multi-sequence feature extraction block, a feature selection block, and a fusion block. The multi-sequence feature extraction block consists of multiple sobel dense conv leaky ReLu block (SDCL-block). SDCL-block mainly realizes the extraction of edge features, shallow features, and deep features. The feature selection block identifies the feature channels with rich texture information and strong discrimination ability through the effective combination of global information entropy criterion and feature jump connection. The feature fusion block mainly comprises multi-head and spatial attention mechanisms, which can realize the fusion of intra-modality and inter-modality features. On this basis, considering the influence of tumor spatial location and structure information on the fusion results, a loss function is designed, which is a weighted combination of texture loss, structure loss, and saliency loss so that texture information from multimodal magnetic resonance imaging (MMRI) and saliency information from different anatomical structures of the brain can be fused at the same time to improve the expression ability of features. In this paper, the TIEF algorithm is trained and validated on the MMRI and (Single-Photon Emission Computed Tomography-MRI) SPECT-MRI datasets of glioma and generalized on the (Computed Tomography-MRI) CT-MRI dataset of meningioma to verify the performance of the TIEF algorithm. In the image fusion task, quantitative results showed that TIEF exhibited optimal or suboptimal performance in information entropy, spatial frequency, and average gradient metrics. Qualitative results indicate that the fused images can highlight tumor and edematous features. A downstream image segmentation task was used for evaluation to further verify TIEF's effectiveness. TIEF achieved the best results in both (Dice similarity coefficient) Dice and (Hausdorff distance 95%) HD95 segmentation metrics. In the generalization task, quantitative results indicated that TIEF obtained more information in the meningioma dataset. In conclusion, TIEF can effectively achieve cross-domain information acquisition and fusion and has robustness and generalization ability.

Mutually enhanced multi-view information learning for segmentation of lung tumor in CT images

Mutual Information-Based Graph Co-Attention Networks for Multimodal Prior-Guided Magnetic Resonance Imaging Segmentation

Multimodal Priors Guided Segmentation of Liver Lesions in MRI Using Mutual Information Based Graph Co-Attention Networks.

Multi-view attribute learning and context relationship encoding enhanced segmentation of lung tumors from CT images

Joint segmentation of tumors in 3D PET-CT images with a network fusing multi-view and multi-modal information

Convolutional bi-directional learning and spatial enhanced attentions for lung tumor segmentation

Semisupervised 3D segmentation of pancreatic tumors in positron emission tomography/computed tomography images using a mutual information minimization and cross-fusion strategy

Integrating cross-modality hallucinated MRI with CT to aid mediastinal lung tumor segmentation

Multi-scale and multi-view network for lung tumor segmentation

Exploiting Partial Common Information Microstructure for Multi-Modal Brain Tumor Segmentation

Uncertainty-Aware Multi-Dimensional Mutual Learning for Brain and Brain Tumor Segmentation

Complementary Information Mutual Learning for Multimodality Medical Image Segmentation

Modality-aware Mutual Learning for Multi-modal Medical Image Segmentation

Multitask connected U-Net: automatic lung cancer segmentation from CT images using PET knowledge guidance

Deep Learning for Variational Multimodality Tumor Segmentation in PET/CT

Multimodal Medical Image Fusion Network Based on Target Information Enhancement

Multimodal Spatial Attention Module for Targeting Multimodal PET-CT Lung Tumor Segmentation

IMIIN: An inter-modality information interaction network for 3D multi-modal breast tumor segmentation

Multi-Scale Segmentation Squeeze-and-Excitation UNet with Conditional Random Field for Segmenting Lung Tumor from CT Images

Multi-phase features interaction transformer network for liver tumor segmentation and microvascular invasion assessment in contrast-enhanced CT

LSAM: L2-norm self-attention and latent space feature interaction for automatic 3D multi-modal head and neck tumor segmentation