Abstract:Denoising Diffusion Probabilistic Models (DDPMs) are a very popular class of deep generative model that have been successfully applied to a diverse range of problems including image and video generation, protein and material synthesis, weather forecasting, and neural surrogates of partial differential equations. Despite their ubiquity it is hard to find an introduction to DDPMs which is simple, comprehensive, clean and clear. The compact explanations necessary in research papers are not able to elucidate all of the different design steps taken to formulate the DDPM and the rationale of the steps that are presented is often omitted to save space. Moreover, the expositions are typically presented from the variational lower bound perspective which is unnecessary and arguably harmful as it obfuscates why the method is working and suggests generalisations that do not perform well in practice. On the other hand, perspectives that take the continuous time-limit are beautiful and general, but they have a high barrier-to-entry as they require background knowledge of stochastic differential equations and probability flow. In this note, we distill down the formulation of the DDPM into six simple steps each of which comes with a clear rationale. We assume that the reader is familiar with fundamental topics in machine learning including basic probabilistic modelling, Gaussian distributions, maximum likelihood estimation, and deep learning.

What problem does this paper attempt to address?

The paper aims to address issues in generative modeling, specifically by transforming complex unsupervised generative modeling problems into a series of simple supervised regression problems. Specifically, the paper achieves this goal through the following points: 1. **Simplified Introduction**: The paper attempts to provide a simple, comprehensive, and clear introduction explaining the basic concepts of Denoising Diffusion Probabilistic Models (DDPM) and their design steps, elucidating the principles behind each step. 2. **Six-Step Method**: The authors break down the construction process of DDPM into 6 simple steps, each with clear principles and related design space. 3. **Data Augmentation Strategy**: The paper proposes a data augmentation strategy, which involves gradually adding noise to create datasets of different fidelities, thereby transforming the generative modeling problem into a supervised learning problem. 4. **Training Objective Function**: A step-by-step supervised objective function is defined for training each regression model, and a parameter-sharing method is proposed to reduce the number of model parameters. 5. **Gaussian Regression Model**: A Gaussian distribution is chosen as the regression model, and this choice is used to reduce Monte Carlo noise in the training objective. 6. **Model Parameterization**: The paper discusses how to parameterize the model using neural networks and introduces two different parameterization methods: predicting clean data (x(0)-parameterization) and estimating noise (ϵ-parameterization). 7. **Simplified and Variational Objectives**: Two popular objective functions are proposed: the simplified DDPM objective and the variational diffusion model objective, and their relationship is explored. 8. **Augmentation Coefficient Selection**: The paper discusses how to choose the augmentation coefficient λ_t, provides several common selection schemes, and points out that in the extreme case, only the signal-to-noise ratio at the beginning and end affects the loss function. Through the above methods, the paper attempts to make the theory of DDPM more understandable and provide guidance for practical applications.

Denoising Diffusion Probabilistic Models in Six Simple Steps

Multi-Step Denoising Scheduled Sampling: Towards Alleviating Exposure Bias for Diffusion Models

Denoising Diffusion Step-aware Models

An overview of diffusion models for generative artificial intelligence

UDPM: Upsampling Diffusion Probabilistic Models

Diffusion Model from Scratch

Denoising Lévy Probabilistic Models

Structured Denoising Diffusion Models in Discrete State-Spaces

Denoising diffusion probabilistic models are optimally adaptive to unknown low dimensionality

Mix-DDPM: Enhancing Diffusion Models Through Fitting Mixture Noise with Global Stochastic Offset

Pseudo Numerical Methods for Diffusion Models on Manifolds

Semi-Implicit Denoising Diffusion Models (SIDDMs)

Directly Denoising Diffusion Models

Enhancing Diffusion Models for High-Quality Image Generation

Filling Deterministic Approximate Gap in Denoising Diffusion Probabilistic Models with Energy

Learning to Discretize Denoising Diffusion ODEs

Steering Masked Discrete Diffusion Models via Discrete Denoising Posterior Prediction

Think While You Generate: Discrete Diffusion with Planned Denoising

Going beyond Compositions, DDPMs Can Produce Zero-Shot Interpolations

Efficient Denoising Diffusion Via Probabilistic Masking

The Missing U for Efficient Diffusion Models