Abstract:Diffusion Probabilistic Models (DPMs) have demonstrated substantial promise in image generation tasks but heavily rely on the availability of large amounts of training data. Previous works, like GANs, have tackled the limited data problem by transferring pre-trained models learned with sufficient data. However, those methods are hard to be utilized in DPMs since the distinct differences between DPM-based and GAN-based methods, showing in the unique iterative denoising process integral and the need for many timesteps with no-targeted noise in DPMs. In this paper, we propose a novel DPMs-based transfer learning method, TAN, to address the limited data problem. It includes two strategies: similarity-guided training, which boosts transfer with a classifier, and adversarial noise selection which adaptive chooses targeted noise based on the input image. Extensive experiments in the context of few-shot image generation tasks demonstrate that our method is not only efficient but also excels in terms of image quality and diversity when compared to existing GAN-based and DDPM-based methods.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to effectively perform transfer learning in diffusion models to generate high - quality and diverse images when the amount of data is limited. Specifically, existing generative models such as GANs have made certain progress in dealing with the problem of limited data, but these methods are difficult to be directly applied to diffusion models (DPMs) because DPMs have a unique iterative denoising process, require a large number of time steps, and the noise is non - targeted, which leads to two main challenges in the transfer process: 1. **Estimation of transfer direction**: Due to the iterative denoising characteristics of DPMs, it is difficult to directly compare the generated blurry images with the target images, which makes it difficult to determine the transfer direction from the source domain to the target domain. 2. **Non - targeted noise**: The noise in DPMs is completely random Gaussian noise and does not depend on the input image, which leads to an unbalanced influence on different images during the training process, thus affecting the speed and effect of transfer. To solve these problems, the author proposes a new method named TAN (Transfer Adversarial Noise), which includes two strategies: 1. **Similarity - guided training**: By using a classifier to estimate the differences between the source domain and the target domain, the training of the model is guided, avoiding the need to directly compare individual images. 2. **Adversarial noise selection**: Dynamically select the noise for the input image to minimize the "worst - case" Gaussian noise that the pre - trained model cannot denoise on the target dataset, reduce the required number of training iterations, and improve the efficiency of transfer learning. Through these two strategies, TAN can effectively transfer the pre - trained DPMs to a new target domain and generate high - quality and diverse images with a small number of samples. Experimental results show that TAN outperforms existing GAN - based and DDPM - based methods in few - sample image generation tasks.

Efficient Transfer Learning in Diffusion Models via Adversarial Noise

Transferring Pretrained Diffusion Probabilistic Models

Adv-BDPM: Adversarial Attack Based on Boundary Diffusion Probability Model.

Diffusion-GAN: Training GANs with Diffusion

Accelerating Diffusion Models via Early Stop of the Diffusion Process

Adv-KD: Adversarial Knowledge Distillation for Faster Diffusion Sampling

Tackling the Generative Learning Trilemma with Denoising Diffusion GANs

Diffusion Model for Generative Image Denoising

AdvDiff: Generating Unrestricted Adversarial Examples using Diffusion Models

Truncated Diffusion Probabilistic Models and Diffusion-based Adversarial Auto-Encoders

Few-shot Image Generation with Diffusion Models

Improving Adversarial Robustness by Contrastive Guided Diffusion Process

Enhancing Diffusion-Based Image Synthesis with Robust Classifier Guidance

Entropy-Driven Sampling and Training Scheme for Conditional Diffusion Generation.

Denoising Diffusion Step-aware Models

Learning to Discretize Denoising Diffusion ODEs

Better Diffusion Models Further Improve Adversarial Training

Latent Denoising Diffusion GAN: Faster sampling, Higher image quality

AdjointDPM: Adjoint Sensitivity Method for Gradient Backpropagation of Diffusion Probabilistic Models

Diffusion Models With Learned Adaptive Noise