Abstract:Diffusion models were initially developed for text-to-image generation and are now being utilized to generate high quality synthetic images. Preceded by GANs, diffusion models have shown impressive results using various evaluation metrics. However, commonly used metrics such as FID and IS are not suitable for determining whether diffusion models are simply reproducing the training images. Here we train StyleGAN and a diffusion model, using BRATS20, BRATS21 and a chest x-ray pneumonia dataset, to synthesize brain MRI and chest x-ray images, and measure the correlation between the synthetic images and all training images. Our results show that diffusion models are more likely to memorize the training images, compared to StyleGAN, especially for small datasets and when using 2D slices from 3D volumes. Researchers should be careful when using diffusion models (and to some extent GANs) for medical imaging, if the final goal is to share the synthetic images.

What problem does this paper attempt to address?

### What problem does this paper attempt to solve? This paper aims to explore whether Diffusion Models are prone to memorizing training data when synthesizing medical images and compare them with Generative Adversarial Networks (GANs). Specifically, the paper focuses on the following points: 1. **The memorization problem of Diffusion Models**: - Diffusion Models were initially used for text - to - image generation tasks and are now widely used to generate high - quality synthetic images. - Although Diffusion Models perform well on multiple evaluation metrics, common evaluation metrics such as Fréchet Inception Distance (FID) and Inception Score (IS) cannot effectively detect whether the model has simply copied the training data. 2. **Privacy issues in medical image synthesis**: - In the field of medical imaging, privacy protection is crucial, especially when dealing with sensitive patient data. - If Diffusion Models are prone to memorizing training data, then the synthetic images generated using these models may leak patients' personal information, violating privacy regulations such as GDPR. 3. **Comparison of the memorization degrees of different generative models**: - The paper trains StyleGAN and Diffusion Models to synthesize brain MRI and chest X - ray images and measures the correlation between the synthetic images and all training images. - The results show that Diffusion Models are more likely to memorize training data than StyleGAN, especially when the dataset is small and when using 2D slices of 3D volumes. 4. **Selection of evaluation methods**: - The paper not only uses the common FID and IS metrics but also introduces pixel - level correlation analysis to more accurately evaluate the memorization degree of the model. - This comprehensive evaluation method can better reflect the performance of generative models in practical applications, especially for the special field of medical image synthesis. ### Main research content - **Datasets**: Use BRATS20, BRATS21 and a pneumonia chest X - ray dataset to train and test models. - **Generative models**: Include StyleGAN and Diffusion Models. - **Evaluation metrics**: Besides FID and IS, also calculate the highest correlation between the synthetic images and the training images. - **Experimental design**: By changing the hyper - parameters of the model (such as the number of trainable parameters), study their impact on memorization. ### Research conclusions - Diffusion Models are indeed more likely to memorize training data when synthesizing medical images, especially when the training dataset is small. - Common metrics such as FID and IS can evaluate the quality of generated images, but cannot accurately reflect the memorization degree of the model. - In medical image synthesis, researchers should carefully select generative models to ensure that patients' privacy information will not be leaked. Through these studies, the paper emphasizes the privacy risks that Diffusion Models may bring in the field of medical image synthesis and provides an important reference for future research.

Beware of diffusion models for synthesizing medical images -- A comparison with GANs in terms of memorizing brain MRI and chest x-ray images

Brain tumor segmentation using synthetic MR images -- A comparison of GANs and diffusion models

Spot the fake lungs: Generating Synthetic Medical Images using Neural Diffusion Models

A New Chapter for Medical Image Generation: The Stable Diffusion Method

Diffusion-Based Approaches in Medical Image Generation and Analysis

Evaluating the feasibility of using Generative Models to generate Chest X-Ray Data

Advanced image generation for cancer using diffusion models

Enabling Competitive Performance of Medical Imaging with Diffusion Model-generated Images without Privacy Leakage

Investigating Data Memorization in 3D Latent Diffusion Models for Medical Image Synthesis

Unconditional Latent Diffusion Models Memorize Patient Imaging Data: Implications for Openly Sharing Synthetic Data

How Good Are Synthetic Medical Images? An Empirical Study with Lung Ultrasound

GANs for Medical Image Synthesis: An Empirical Study

Synthetic Brain Images: Bridging the Gap in Brain Mapping With Generative Adversarial Model

Overcoming barriers to data sharing with medical image generation: a comprehensive evaluation

A multimodal comparison of latent denoising diffusion probabilistic models and generative adversarial networks for medical image synthesis

XReal: Realistic Anatomy and Pathology-Aware X-ray Generation via Controllable Diffusion Model

Denoising diffusion probabilistic models for 3D medical image generation

2D medical image synthesis using transformer-based denoising diffusion probabilistic model

A Critical Assessment of Generative Models for Synthetic Data Augmentation on Limited Pneumonia X-ray Data

Enhancing Medical Imaging with GANs Synthesizing Realistic Images from Limited Data

Brain tumor image generation using an aggregation of GAN models with style transfer