Abstract:Abstract Purpose This study proposed an end-to-end unsupervised medical fusion generative adversarial network, MedFusionGAN, to fuse computed tomography (CT) and high-resolution isotropic 3D T1-Gd Magnetic resonance imaging (MRI) image sequences to generate an image with CT bone structure and MRI soft tissue contrast to improve target delineation and to reduce the radiotherapy planning time. Methods We used a publicly available multicenter medical dataset (GLIS-RT, 230 patients) from the Cancer Imaging Archive. To improve the models generalization, we consider different imaging protocols and patients with various brain tumor types, including metastases. The proposed MedFusionGAN consisted of one generator network and one discriminator network trained in an adversarial scenario. Content, style, and L1 losses were used for training the generator to preserve the texture and structure information of the MRI and CT images. Results The MedFusionGAN successfully generates fused images with MRI soft-tissue and CT bone contrast. The results of the MedFusionGAN were quantitatively and qualitatively compared with seven traditional and eight deep learning (DL) state-of-the-art methods. Qualitatively, our method fused the source images with the highest spatial resolution without adding the image artifacts. We reported nine quantitative metrics to quantify the preservation of structural similarity, contrast, distortion level, and image edges in fused images. Our method outperformed both traditional and DL methods on six out of nine metrics. And it got the second performance rank for three and two quantitative metrics when compared with traditional and DL methods, respectively. To compare soft-tissue contrast, intensity profile along tumor and tumor contours of the fusion methods were evaluated. MedFusionGAN provides a more consistent, better intensity profile, and a better segmentation performance. Conclusions The proposed end-to-end unsupervised method successfully fused MRI and CT images. The fused image could improve targets and OARs delineation, which is an important aspect of radiotherapy treatment planning.

Unsupervised Fusion of Misaligned PAT and MRI Images via Mutually Reinforcing Cross-Modality Image Generation and Registration

Image Fusion of CT and MR with Sparse Representation in NSST Domain

Performance of Medical Image Fusion in High-level Analysis Tasks: A Mutual Enhancement Framework for Unaligned PAT and MRI Image Fusion

High-Quality Fusion and Visualization for MR-PET Brain Tumor Images via Multi-Dimensional Features

Three-Dimensional Medical Image Fusion with Deformable Cross-Attention

Medical Image Fusion for Multiple Diseases Features Enhancement

Multi-modality MRI fusion with patch complementary pre-training for internet of medical things-based smart healthcare

Medical Image Segmentation Based on Multi-Modal Convolutional Neural Network: Study on Image Fusion Schemes

Medical Image Fusion with Parameter-Adaptive Pulse Coupled Neural Network in Nonsubsampled Shearlet Transform Domain

MRI and PET/SPECT fusion with pseudo spatial frequency in local laplacian filter domain

MURF: Mutually Reinforcing Multi-Modal Image Registration and Fusion

MedFusionGAN: multimodal medical image fusion using an unsupervised deep generative adversarial network

FMB: Dual-view fusion and registration of 2D DSA images and 3D MRA images for neurointerventional-based procedures

Cross-contrast mutual fusion network for joint MRI reconstruction and super-resolution

Fusion of mis-registered GFP and phase contrast images with convolutional sparse representation and adaptive region energy rule.

Adaptive Multimodal Image Fusion with a Deep Pyramidal Residual Learning Network

Multimodal MRI Volumetric Data Fusion With Convolutional Neural Networks

Multimodal Medical Image Fusion of Positron Emission Tomography and Magnetic Resonance Imaging Using Generative Adversarial Networks

An Effective Multimodal Image Fusion Method Using MRI and PET for Alzheimer's Disease Diagnosis

FEFA: Frequency Enhanced Multi-Modal MRI Reconstruction with Deep Feature Alignment

MMFGAN: A novel multimodal brain medical image fusion based on the improvement of generative adversarial network