Theoretical research on generative diffusion models: an overview

Melike Nur Yeğin,Mehmet Fatih Amasyalı

2024-04-13

Abstract:Generative diffusion models showed high success in many fields with a powerful theoretical background. They convert the data distribution to noise and remove the noise back to obtain a similar distribution. Many existing reviews focused on the specific application areas without concentrating on the research about the algorithm. Unlike them we investigated the theoretical developments of the generative diffusion models. These approaches mainly divide into two: training-based and sampling-based. Awakening to this allowed us a clear and understandable categorization for the researchers who will make new developments in the future.

Machine Learning,Artificial Intelligence,Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is a review of theoretical research on generative diffusion models (GDMs). Specifically, the paper focuses on the following aspects:

1. **Brief Explanation of Existing Generative Models**: The paper first briefly introduces the existing generative models and discusses why diffusion models are needed. This includes generative adversarial networks (GANs), variational auto - encoders (VAEs), flow - based models, autoregressive models, and energy - based models.

2. **Core Research on Diffusion Models**: The paper systematically reviews the core research on diffusion models, explaining the relationships between these studies and their shortcomings. It mainly discusses three core literatures:
- **Denoising Diffusion Probabilistic Models (DDPM)**: Proposed by Ho et al., inspired by non - equilibrium thermodynamics theory, it uses latent variables to estimate probability distributions.
- **Noise Conditional Score Networks (NCSN)**: Proposed by Song and Ermon, it estimates the data distribution score function at different noise levels by training a shared network.
- **Score - based Modeling with Stochastic Differential Equations (Score SDE)**: Proposed by Song et al., it solves the diffusion process through forward and backward SDEs, providing a general framework that unifies DDPM and NCSN.

3. **Classification of Theoretical Research**: The paper classifies the theoretical research on diffusion models according to the research focus into two major categories: training methods and sampling methods, and further subdivides them under these categories.

4. **Evaluation Metrics and Benchmark Results**: The paper explains the evaluation metrics of diffusion models and provides benchmark results on common datasets.

5. **Current Research Status and Future Directions**: The paper discusses the current research status in the field of diffusion models and points out future research directions.

In summary, this paper aims to comprehensively overview the theoretical development of generative diffusion models, providing researchers with a clear understanding framework for new research and development in the future.