Abstract:Diffusion models excel in generating images that closely resemble their training data but are also susceptible to data memorization, raising privacy, ethical, and legal concerns, particularly in sensitive domains such as medical imaging. We hypothesize that this memorization stems from the overparameterization of deep models and propose that regularizing model capacity during fine-tuning can mitigate this issue. Firstly, we empirically show that regulating the model capacity via Parameter-efficient fine-tuning (PEFT) mitigates memorization to some extent, however, it further requires the identification of the exact parameter subsets to be fine-tuned for high-quality generation. To identify these subsets, we introduce a bi-level optimization framework, MemControl, that automates parameter selection using memorization and generation quality metrics as rewards during fine-tuning. The parameter subsets discovered through MemControl achieve a superior tradeoff between generation quality and memorization. For the task of medical image generation, our approach outperforms existing state-of-the-art memorization mitigation strategies by fine-tuning as few as 0.019% of model parameters. Moreover, we demonstrate that the discovered parameter subsets are transferable to non-medical domains. Our framework is scalable to large datasets, agnostic to reward functions, and can be integrated with existing approaches for further memorization mitigation. To the best of our knowledge, this is the first study to empirically evaluate memorization in medical images and propose a targeted yet universal mitigation strategy. The code is available at <a class="link-external link-https" href="https://github.com/Raman1121/Diffusion_Memorization_HPO" rel="external noopener nofollow">this https URL</a>

An Inversion-based Measure of Memorization for Diffusion Models

Detecting, Explaining, and Mitigating Memorization in Diffusion Models

Unveiling and Mitigating Memorization in Text-to-image Diffusion Models through Cross Attention

On Memorization in Diffusion Models

Memorized Images in Diffusion Models share a Subspace that can be Located and Deleted

Unveiling Structural Memorization: Structural Membership Inference Attack for Text-to-Image Diffusion Models

Investigating Memorization in Video Diffusion Models

Towards Memorization-Free Diffusion Models

MemBench: Memorized Image Trigger Prompt Dataset for Diffusion Models

Towards a Theoretical Understanding of Memorization in Diffusion Models

Understanding (Un)Intended Memorization in Text-to-Image Generative Models

Iterative Ensemble Training with Anti-Gradient Control for Mitigating Memorization in Diffusion Models

Memory Triggers: Unveiling Memorization in Text-To-Image Generative Models through Word-Level Duplication

Investigating Data Memorization in 3D Latent Diffusion Models for Medical Image Synthesis

Finding NeMo: Localizing Neurons Responsible For Memorization in Diffusion Models

Embedding Space Selection for Detecting Memorization and Fingerprinting in Generative Models

DIAGNOSIS: Detecting Unauthorized Data Usages in Text-to-image Diffusion Models

MemControl: Mitigating Memorization in Diffusion Models via Automated Parameter Selection

Classifier-Free Guidance inside the Attraction Basin May Cause Memorization

Real-World Benchmarks Make Membership Inference Attacks Fail on Diffusion Models