Abstract:Abstract Purpose This study proposed an end-to-end unsupervised medical fusion generative adversarial network, MedFusionGAN, to fuse computed tomography (CT) and high-resolution isotropic 3D T1-Gd Magnetic resonance imaging (MRI) image sequences to generate an image with CT bone structure and MRI soft tissue contrast to improve target delineation and to reduce the radiotherapy planning time. Methods We used a publicly available multicenter medical dataset (GLIS-RT, 230 patients) from the Cancer Imaging Archive. To improve the models generalization, we consider different imaging protocols and patients with various brain tumor types, including metastases. The proposed MedFusionGAN consisted of one generator network and one discriminator network trained in an adversarial scenario. Content, style, and L1 losses were used for training the generator to preserve the texture and structure information of the MRI and CT images. Results The MedFusionGAN successfully generates fused images with MRI soft-tissue and CT bone contrast. The results of the MedFusionGAN were quantitatively and qualitatively compared with seven traditional and eight deep learning (DL) state-of-the-art methods. Qualitatively, our method fused the source images with the highest spatial resolution without adding the image artifacts. We reported nine quantitative metrics to quantify the preservation of structural similarity, contrast, distortion level, and image edges in fused images. Our method outperformed both traditional and DL methods on six out of nine metrics. And it got the second performance rank for three and two quantitative metrics when compared with traditional and DL methods, respectively. To compare soft-tissue contrast, intensity profile along tumor and tumor contours of the fusion methods were evaluated. MedFusionGAN provides a more consistent, better intensity profile, and a better segmentation performance. Conclusions The proposed end-to-end unsupervised method successfully fused MRI and CT images. The fused image could improve targets and OARs delineation, which is an important aspect of radiotherapy treatment planning.

BTMF-GAN: A Multi-Modal MRI Fusion Generative Adversarial Network for Brain Tumors

MMFGAN: A novel multimodal brain medical image fusion based on the improvement of generative adversarial network

MedFusionGAN: multimodal medical image fusion using an unsupervised deep generative adversarial network

Multimodal Medical Image Fusion of Positron Emission Tomography and Magnetic Resonance Imaging Using Generative Adversarial Networks

Multi -Modality Brain Tumor Segmentation Network Based on Collaborative Feature Fusion

MM-BiFPN: Multi-Modality Fusion Network With Bi-FPN for MRI Brain Tumor Segmentation

A multi-modality fusion network based on attention mechanism for brain tumor segmentation

Feature-enhanced generation and multi-modality fusion based deep neural network for brain tumor segmentation with missing MR modalities

Modality-level cross-connection and attentional feature fusion based deep neural network for multi-modal brain tumor segmentation

CT and MRI Image Fusion via Coupled Feature-Learning GAN

An Attentive Multi-Modal CNN for Brain Tumor Radiogenomic Classification

Multi-Modality Generative Adversarial Networks with Tumor Consistency Loss for Brain MR Image Synthesis

[CT and MRI fusion based on generative adversarial network and convolutional neural networks under image enhancement].

Self-Supervised Multi-Modal Hybrid Fusion Network for Brain Tumor Segmentation.

Multi-modal fusion deep transfer learning for accurate brain tumor classification using magnetic resonance imaging images

Multi-modal Brain Tumor Segmentation via Missing Modality Synthesis and Modality-level Attention Fusion

Brain Tumor Segmentation in Multimodal MRI Via Pixel-Level and Feature-Level Image Fusion.

MCMT-GAN: Multi-Task Coherent Modality Transferable GAN for 3D Brain Image Synthesis.

A Generative Adversarial Network For Medical Image Fusion

DualMMP-GAN: Dual-scale Multi-Modality Perceptual Generative Adversarial Network for Medical Image Segmentation

Multi-Modal MRI Image Synthesis via GAN With Multi-Scale Gate Mergence