Abstract:PurposeComplementary information obtained from multiple contrasts of tissue facilitates physicians assessing, diagnosing and planning treatment of a variety of diseases. However, acquiring multiple contrasts magnetic resonance images (MRI) for every patient using multiple pulse sequences is time‐consuming and expensive, where, medical image synthesis has been demonstrated as an effective alternative. The purpose of this study is to develop a unified framework for multimodal MR image synthesis.MethodsA unified generative adversarial network consisting of only a single generator and a single discriminator was developed to learn the mappings among images of four different modalities. The generator took an image and its modality label as inputs and learned to synthesize the image in the target modality, while the discriminator was trained to distinguish between real and synthesized images and classify them to their corresponding modalities. The network was trained and tested using multimodal brain MRI consisting of four different contrasts which are T1‐weighted (T1), T1‐weighted and contrast‐enhanced (T1c), T2‐weighted (T2), and fluid‐attenuated inversion recovery (Flair). Quantitative assessments of our proposed method were made through computing normalized mean absolute error (NMAE), peak signal‐to‐noise ratio (PSNR), structural similarity index measurement (SSIM), visual information fidelity (VIF), and naturalness image quality evaluator (NIQE).ResultsThe proposed model was trained and tested on a cohort of 274 glioma patients with well‐aligned multi‐types of MRI scans. After the model was trained, tests were conducted by using each of T1, T1c, T2, Flair as a single input modality to generate its respective rest modalities. Our proposed method shows high accuracy and robustness for image synthesis with arbitrary MRI modality that is available in the database as input. For example, with T1 as input modality, the NMAEs for the generated T1c, T2, Flair respectively are 0.034 ± 0.005, 0.041 ± 0.006, and 0.041 ± 0.006, the PSNRs respectively are 32.353 ± 2.525 dB, 30.016 ± 2.577 dB, and 29.091 ± 2.795 dB, the SSIMs are 0.974 ± 0.059, 0.969 ± 0.059, and 0.959 ± 0.059, the VIF are 0.750 ± 0.087, 0.706 ± 0.097, and 0.654 ± 0.062, and NIQE are 1.396 ± 0.401, 1.511 ± 0.460, and 1.259 ± 0.358, respectively.ConclusionsWe proposed a novel multimodal MR image synthesis method based on a unified generative adversarial network. The network takes an image and its modality label as inputs and synthesizes multimodal images in a single forward pass. The results demonstrate that the proposed method is able to accurately synthesize multimodal MR images from a single MR image.

A layer-wise fusion network incorporating self-supervised learning for multimodal MR image synthesis

Hi-Net: Hybrid-fusion Network for Multi-modal MR Image Synthesis

Multimodal MRI Synthesis Using Unified Generative Adversarial Networks.

Multimodal MR Image Synthesis Using Gradient Prior and Adversarial Learning

An hetero-modal deep learning framework for medical image synthesis applied to contrast and non-contrast MRI

Coarse-to-Fine Learning Framework for Semi-supervised Multimodal MRI Synthesis

MSAIF-Net: A Multistage Spatial Attention-Based Invertible Fusion Network for MR Images.

Research on Dual-adversarial MR Image Fusion Network Using Pre-trained Model for Feature Extraction

SS-SSAN: a Self-Supervised Subspace Attentional Network for Multi-Modal Medical Image Fusion

Convolutional neural network with coarse-to-fine resolution fusion and residual learning structures for cross-modality image synthesis

Learning Unified Hyper-Network for Multi-Modal MR Image Synthesis and Tumor Segmentation With Missing Modalities

Multi-Modality MR Image Synthesis via Confidence-Guided Aggregation and Cross-Modality Refinement

SLMT-Net: A Self-supervised Learning Based Multi-scale Transformer Network for Cross-Modality MR Image Synthesis

SIMFusion: A Semantic Information-Guided Modality-Specific Fusion Network for MR Images

An Attention-based Multi-Scale Feature Learning Network for Multimodal Medical Image Fusion

Multimodal MRI Volumetric Data Fusion With Convolutional Neural Networks

Multi-scale Transformer Network with Edge-aware Pre-training for Cross-Modality MR Image Synthesis

MLMFNet: A Multi-Level Modality Fusion Network for Multi-Modal Accelerated MRI Reconstruction

Unified Multi-Modal Image Synthesis for Missing Modality Imputation

MMFGAN: A novel multimodal brain medical image fusion based on the improvement of generative adversarial network

M4FNet: Multimodal Medical Image Fusion Network Via Multi-Receptive-field and Multi-Scale Feature Integration.