Towards Memorization-Free Diffusion Models

Chen Chen,Daochang Liu,Chang Xu

2024-04-01

Abstract:Pretrained diffusion models and their outputs are widely accessible due to their exceptional capacity for synthesizing high-quality images and their open-source nature. The users, however, may face litigation risks owing to the models' tendency to memorize and regurgitate training data during inference. To address this, we introduce Anti-Memorization Guidance (AMG), a novel framework employing three targeted guidance strategies for the main causes of memorization: image and caption duplication, and highly specific user prompts. Consequently, AMG ensures memorization-free outputs while maintaining high image quality and text alignment, leveraging the synergy of its guidance methods, each indispensable in its own right. AMG also features an innovative automatic detection system for potential memorization during each step of inference process, allows selective application of guidance strategies, minimally interfering with the original sampling process to preserve output utility. We applied AMG to pretrained Denoising Diffusion Probabilistic Models (DDPM) and Stable Diffusion across various generation tasks. The results demonstrate that AMG is the first approach to successfully eradicates all instances of memorization with no or marginal impacts on image quality and text-alignment, as evidenced by FID and CLIP scores.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper addresses the potential problem of training data memorization in the diffusion model during the inference process, where the model may replicate and output the training data. To solve this problem, the paper proposes a framework called Anti-Memorization Guidance (AMG), which includes three guiding strategies: generalization guidance, title deduplication guidance, and distinctiveness guidance, aiming to respectively address the issues of image and title duplication as well as memory phenomena caused by high specificity of user prompts. AMG also has an automatic detection system that selectively applies guiding strategies during the inference process, minimizing interference with the original sampling process while maintaining high image quality and text alignment. Experimental results show that AMG successfully eliminates all memory instances with minimal impact on image quality and text alignment.