Abstract:Typical generative diffusion models rely on a Gaussian diffusion process for training the backward transformations, which can then be used to generate samples from Gaussian noise. However, real world data often takes place in discrete-state spaces, including many scientific applications. Here, we develop a theoretical formulation for arbitrary discrete-state Markov processes in the forward diffusion process using exact (as opposed to variational) analysis. We relate the theory to the existing continuous-state Gaussian diffusion as well as other approaches to discrete diffusion, and identify the corresponding reverse-time stochastic process and score function in the continuous-time setting, and the reverse-time mapping in the discrete-time setting. As an example of this framework, we introduce ``Blackout Diffusion'', which learns to produce samples from an empty image instead of from noise. Numerical experiments on the CIFAR-10, Binarized MNIST, and CelebA datasets confirm the feasibility of our approach. Generalizing from specific (Gaussian) forward processes to discrete-state processes without a variational approximation sheds light on how to interpret diffusion models, which we discuss.

What problem does this paper attempt to address?

The paper mainly addresses the following issues: 1. **Construction of Theoretical Framework**: The paper proposes an exact theoretical framework applicable to any discrete-state Markov process, including both discrete-time and continuous-time cases. This framework enables researchers to use diffusion models to handle non-continuous data. 2. **Discrete-State Diffusion Model**: In response to the problem that existing diffusion models are primarily based on Gaussian processes and applied to continuous domain data, the paper develops a diffusion model theory suitable for discrete state spaces, with particular attention to cases that do not require variational approximations. 3. **Blackout Diffusion Model**: The paper introduces a specific model instance called "Blackout Diffusion," which generates samples from blank images (rather than noise) and is suitable for discrete datasets such as digital images. This model is trained in a pure death process and can effectively generate samples for image datasets (e.g., CIFAR-10, Binarized MNIST, and CelebA). 4. **Relationship with Existing Methods**: The paper compares the proposed theory with existing continuous-state Gaussian diffusion models and other discrete-state diffusion methods, exploring their connections and differences. 5. **Reverse Process and Loss Function**: The paper also proposes a reverse process and corresponding loss function for discrete states, which helps train the model to generate data in reverse. These theoretical results can be seen as discrete-state counterparts to the work of Anderson et al. on Itô stochastic differential equations. In summary, the main contribution of this paper lies in providing a general and precise theoretical framework for discrete-state diffusion models and demonstrating its feasibility in practical applications by introducing the specific model of Blackout Diffusion.

Blackout Diffusion: Generative Diffusion Models in Discrete-State Spaces

Where to Diffuse, How to Diffuse, and How to Get Back: Automated Learning for Multivariate Diffusions

Diffusion Models for Graphs Benefit From Discrete State Spaces

Diffusion with Forward Models: Solving Stochastic Inverse Problems Without Direct Supervision

GUD: Generation with Unified Diffusion

Unifying Diffusion Models' Latent Space, with Applications to CycleDiffusion and Guidance

Non-Normal Diffusion Models

From Denoising Diffusions to Denoising Markov Models

Diffusion Models Generate Images Like Painters: an Analytical Theory of Outline First, Details Later

Generalized Diffusion Model with Adjusted Offset Noise

Multilevel Diffusion: Infinite Dimensional Score-Based Diffusion Models for Image Generation

Simultaneous Image-to-Zero and Zero-to-Noise: Diffusion Models with Analytical Image Attenuation

Latent Abstractions in Generative Diffusion Models

Convergence Analysis of Discrete Diffusion Model: Exact Implementation through Uniformization

An Overview of Diffusion Models: Applications, Guided Generation, Statistical Rates and Optimization

Simplified and Generalized Masked Diffusion for Discrete Data

Discrete generative diffusion models without stochastic differential equations: a tensor network approach

Nested Diffusion Processes for Anytime Image Generation

Discrete Modeling via Boundary Conditional Diffusion Processes

Critical windows: non-asymptotic theory for feature emergence in diffusion models