Abstract:PurposeComplementary information obtained from multiple contrasts of tissue facilitates physicians assessing, diagnosing and planning treatment of a variety of diseases. However, acquiring multiple contrasts magnetic resonance images (MRI) for every patient using multiple pulse sequences is time‐consuming and expensive, where, medical image synthesis has been demonstrated as an effective alternative. The purpose of this study is to develop a unified framework for multimodal MR image synthesis.MethodsA unified generative adversarial network consisting of only a single generator and a single discriminator was developed to learn the mappings among images of four different modalities. The generator took an image and its modality label as inputs and learned to synthesize the image in the target modality, while the discriminator was trained to distinguish between real and synthesized images and classify them to their corresponding modalities. The network was trained and tested using multimodal brain MRI consisting of four different contrasts which are T1‐weighted (T1), T1‐weighted and contrast‐enhanced (T1c), T2‐weighted (T2), and fluid‐attenuated inversion recovery (Flair). Quantitative assessments of our proposed method were made through computing normalized mean absolute error (NMAE), peak signal‐to‐noise ratio (PSNR), structural similarity index measurement (SSIM), visual information fidelity (VIF), and naturalness image quality evaluator (NIQE).ResultsThe proposed model was trained and tested on a cohort of 274 glioma patients with well‐aligned multi‐types of MRI scans. After the model was trained, tests were conducted by using each of T1, T1c, T2, Flair as a single input modality to generate its respective rest modalities. Our proposed method shows high accuracy and robustness for image synthesis with arbitrary MRI modality that is available in the database as input. For example, with T1 as input modality, the NMAEs for the generated T1c, T2, Flair respectively are 0.034 ± 0.005, 0.041 ± 0.006, and 0.041 ± 0.006, the PSNRs respectively are 32.353 ± 2.525 dB, 30.016 ± 2.577 dB, and 29.091 ± 2.795 dB, the SSIMs are 0.974 ± 0.059, 0.969 ± 0.059, and 0.959 ± 0.059, the VIF are 0.750 ± 0.087, 0.706 ± 0.097, and 0.654 ± 0.062, and NIQE are 1.396 ± 0.401, 1.511 ± 0.460, and 1.259 ± 0.358, respectively.ConclusionsWe proposed a novel multimodal MR image synthesis method based on a unified generative adversarial network. The network takes an image and its modality label as inputs and synthesizes multimodal images in a single forward pass. The results demonstrate that the proposed method is able to accurately synthesize multimodal MR images from a single MR image.

Macro2Micro: Cross-modal Magnetic Resonance Imaging Synthesis Leveraging Multi-scale Brain Structures

GAN-MAT: Generative Adversarial Network-based Microstructural Profile Covariance Analysis Toolbox

MouseGAN++: Unsupervised Disentanglement and Contrastive Representation for Multiple MRI Modalities Synthesis and Structural Segmentation of Mouse Brain

Synthetic Brain Images: Bridging the Gap in Brain Mapping With Generative Adversarial Model

MCMT-GAN: Multi-Task Coherent Modality Transferable GAN for 3D Brain Image Synthesis.

Deep Non-Linear Embedding Deformation Network for Cross-Modal Brain MRI Synthesis.

Lesion Mask-based Simultaneous Synthesis of Anatomic and MolecularMR Images using a GAN

Style transfer between Microscopy and Magnetic Resonance Imaging via Generative Adversarial Network in small sample size settings

Multimodal MRI Synthesis Using Unified Generative Adversarial Networks.

Multi-Modal MRI Image Synthesis via GAN With Multi-Scale Gate Mergence

Multi-contrast high-field quality image synthesis for portable low-field MRI using generative adversarial networks and paired data

Multi-Constraint Transferable Generative Adversarial Networks for Cross-Modal Brain Image Synthesis

A Deep Learning Model for Multi-Domain MRI Synthesis Using Generative Adversarial Networks

Metadata-conditioned generative models to synthesize anatomically-plausible 3D brain MRIs

Multiscale Multimodal Medical Imaging - First International Workshop, MMMI 2019, Held in Conjunction with MICCAI 2019, Shenzhen, China, October 13, 2019, Proceedings

Cross2SynNet: cross-device-cross-modal synthesis of routine brain MRI sequences from CT with brain lesion

Enhanced Multiscale Human Brain Imaging by Semi-supervised Digital Staining and Serial Sectioning Optical Coherence Tomography

Harmonizing 1.5T/3T Diffusion Weighted MRI through Development of Deep Learning Stabilized Microarchitecture Estimators

MRI to PET Cross-Modality Translation using Globally and Locally Aware GAN (GLA-GAN) for Multi-Modal Diagnosis of Alzheimer's Disease

Deep learning-based 3D MRI contrast-enhanced synthesis from a 2D noncontrast T2Flair sequence

A Unified Framework for Synthesizing Multisequence Brain MRI via Hybrid Fusion