Abstract:Abstract Purpose This study proposed an end-to-end unsupervised medical fusion generative adversarial network, MedFusionGAN, to fuse computed tomography (CT) and high-resolution isotropic 3D T1-Gd Magnetic resonance imaging (MRI) image sequences to generate an image with CT bone structure and MRI soft tissue contrast to improve target delineation and to reduce the radiotherapy planning time. Methods We used a publicly available multicenter medical dataset (GLIS-RT, 230 patients) from the Cancer Imaging Archive. To improve the models generalization, we consider different imaging protocols and patients with various brain tumor types, including metastases. The proposed MedFusionGAN consisted of one generator network and one discriminator network trained in an adversarial scenario. Content, style, and L1 losses were used for training the generator to preserve the texture and structure information of the MRI and CT images. Results The MedFusionGAN successfully generates fused images with MRI soft-tissue and CT bone contrast. The results of the MedFusionGAN were quantitatively and qualitatively compared with seven traditional and eight deep learning (DL) state-of-the-art methods. Qualitatively, our method fused the source images with the highest spatial resolution without adding the image artifacts. We reported nine quantitative metrics to quantify the preservation of structural similarity, contrast, distortion level, and image edges in fused images. Our method outperformed both traditional and DL methods on six out of nine metrics. And it got the second performance rank for three and two quantitative metrics when compared with traditional and DL methods, respectively. To compare soft-tissue contrast, intensity profile along tumor and tumor contours of the fusion methods were evaluated. MedFusionGAN provides a more consistent, better intensity profile, and a better segmentation performance. Conclusions The proposed end-to-end unsupervised method successfully fused MRI and CT images. The fused image could improve targets and OARs delineation, which is an important aspect of radiotherapy treatment planning.

UCP 2 -ACGAN: An adaptive condition GAN guided by U-shaped context perceptual processor for PET/CT images fusion

Synthesizing Contrast-enhanced Computed Tomography Images with an Improved Conditional Generative Adversarial Network

OA-GAN: Organ-Aware Generative Adversarial Network for Synthesizing Contrast-Enhanced Medical Images

Hybrid Multimodal Medical Image Fusion Method Based on LatLRR and ED-D2GAN

AIGAN: Attention-encoding Integrated Generative Adversarial Network for the Reconstruction of Low-Dose CT and Low-Dose PET Images

CT and MRI Image Fusion via Coupled Feature-Learning GAN

Pixel-Feature Hybrid Fusion for PET/CT Images

Fusion-UDCGAN: Multifocus Image Fusion via a U-Type Densely Connected Generation Adversarial Network

Multiscale Deep-Learning Network Based Reconstruction of PET Images from Sinogram Domain

Segmentation Guided Crossing Dual Decoding Generative Adversarial Network for Synthesizing Contrast-Enhanced Computed Tomography Images

TCGAN: a transformer-enhanced GAN for PET synthetic CT

Cross-Modality Generation of Amyloid PET from FDG PET for Alzheimer’s Disease Diagnosis

3D multi-modality Transformer-GAN for high-quality PET reconstruction

[CT and MRI fusion based on generative adversarial network and convolutional neural networks under image enhancement].

DDcGAN: A Dual-Discriminator Conditional Generative Adversarial Network for Multi-Resolution Image Fusion

A Two-Branch Neural Network for Short-Axis PET Image Quality Enhancement

GMRE-iUnet: Isomorphic Unet fusion model for PET and CT lung tumor images

A Dual-Encoder-Single-Decoder Based Low-dose CT Denoising Network

MedFusionGAN: multimodal medical image fusion using an unsupervised deep generative adversarial network

MMFGAN: A novel multimodal brain medical image fusion based on the improvement of generative adversarial network

AT-GAN: A generative adversarial network with attention and transition for infrared and visible image fusion