Abstract:Diffusion-based generative models demonstrate a transition from memorizing the training dataset to a non-memorization regime as the size of the training set increases. Here, we begin by introducing a mathematically precise definition of this transition in terms of a relative distance: the model is said to be in the non-memorization/`generalization' regime if the generated distribution is almost surely far from the probability distribution associated with a Gaussian kernel approximation to the training dataset, relative to the sampling distribution. Then, we develop an analytically tractable diffusion model and establish a lower bound on Kullback-Leibler divergence between the generated and sampling distribution. The model also features the transition, according to our definition in terms of the relative distance, when the training data is sampled from an isotropic Gaussian distribution. Further, our study reveals that this transition occurs when the individual distance between the generated and underlying sampling distribution begins to decrease with the addition of more training samples. This is to be contrasted with an alternative scenario, where the model's memorization performance degrades, but generalization performance doesn't improve. We also provide empirical evidence indicating that realistic diffusion models exhibit the same alignment of scales.
What problem does this paper attempt to address?
### Problems the paper attempts to solve
This paper mainly explores the transition process from memorizing training data to non - memorizing (i.e., generalization) when diffusion - based generative models generate data. Specifically, the paper aims to answer the following questions:
1. **Nature of the transition from memory to generalization**:
- When the size of the training data set is small, diffusion models tend to memorize the training data.
- As the size of the training data set increases, the model gradually transitions from memorizing data to being able to generate new samples, and this process is called the transition from memory to generalization.
- What are the specific properties of this transition? How can this transition be defined?
2. **Mathematical definition**:
- The paper proposes a mathematical definition based on relative distance to describe this transition. Specifically, two key distances are defined:
- \( E_{TG} \): The distance between the training data distribution and the generated data distribution.
- \( E_{OG} \): The distance between the original data distribution and the generated data distribution.
- The model is considered to be in a non - memory/generalization state if the probability that \( \Delta = E_{TG}-E_{OG}>0 \) is close to 1.
3. **Theoretical framework**:
- The paper constructs a linear diffusion model and derives the lower bound of the Kullback - Leibler divergence between the generated distribution and the sampling distribution.
- Through analysis, it is proved that when the training data comes from an isotropic Gaussian distribution, the model indeed experiences a transition from memory to non - memory.
4. **Empirical research**:
- The paper provides empirical evidence showing that real - world diffusion models also exhibit a similar scale - alignment phenomenon.
- Through experiments, it is verified that when the size of the training data set increases, the memory performance of the model decreases while the generalization performance improves.
### Main contributions
1. **Optimal Gaussian kernel estimation in high - dimensional statistics**:
- Given a finite number of samples, the true distribution can be approximated by the Gaussian kernel with the optimal L2 distance.
- It is found that the variance of the Gaussian kernel coincides with the mixing time of the Ornstein - Uhlenbeck forward diffusion process.
2. **Accurate measurement of the transition from memory to non - memory**:
- A new mathematical measure is proposed to characterize the transition from memory to non - memory.
3. **Analytically solvable diffusion model**:
- An analytically solvable diffusion model is constructed, and the lower bound of the Kullback - Leibler divergence between the generated distribution and the sampling distribution is established.
4. **Transition characteristics of the model**:
- It is proved that when the size of the training data set increases, the transition from memory to non - memory occurs at the moment when the generalization error begins to decrease.
5. **Hypothesis testing**:
- The hypothesis that the transition from memory to non - memory is consistent with the moment when the generalization error begins to decrease is a general characteristic of diffusion models, and it is verified on the real - world U - Net - based nonlinear diffusion model.
### Related work
- The origin of diffusion models can be traced back to early works, such as [3].
- Subsequently, diffusion models have been extended and applied to large - scale image generation tasks, such as DALL - E [6], Stable Diffusion [7], etc.
- Some studies have explored the memory ability of diffusion models [10, 11], but the question of whether the model truly generalizes when not memorizing still needs further research.
Through these contributions, the paper provides a theoretical basis for understanding the transition from memory to generalization in diffusion models and provides guidance for model selection and optimization in practical applications.