Abstract:Diffusion models, which employ stochastic differential equations to sample images through integrals, have emerged as a dominant class of generative models. However, the rationality of the diffusion process itself receives limited attention, leaving the question of whether the problem is well-posed and well-conditioned. In this paper, we uncover a vexing propensity of diffusion models: they frequently exhibit the infinite Lipschitz near the zero point of timesteps. We provide theoretical proofs to illustrate the presence of infinite Lipschitz constants and empirical results to confirm it. The Lipschitz singularities pose a threat to the stability and accuracy during both the training and inference processes of diffusion models. Therefore, the mitigation of Lipschitz singularities holds great potential for enhancing the performance of diffusion models. To address this challenge, we propose a novel approach, dubbed E-TSDM, which alleviates the Lipschitz singularities of the diffusion model near the zero point. Remarkably, our technique yields a substantial improvement in performance. Moreover, as a byproduct of our method, we achieve a dramatic reduction in the Fréchet Inception Distance of acceleration methods relying on network Lipschitz, including DDIM and DPM-Solver, by over 33\%. Extensive experiments on diverse datasets validate our theory and method. Our work may advance the understanding of the general diffusion process, and also provide insights for the design of diffusion models.

Mathematical analysis of singularities in the diffusion model under the submanifold assumption

Lipschitz singularities in diffusion models

Eliminating Lipschitz Singularities in Diffusion Models

Scaling Riemannian Diffusion Models

Convergence of Diffusion Models Under the Manifold Hypothesis in High-Dimensions

Subsampling Error in Stochastic Gradient Langevin Diffusions

A Geometric Perspective on Diffusion Models

Convergence Analysis of Discrete Diffusion Model: Exact Implementation through Uniformization

Tackling the Singularities at the Endpoints of Time Intervals in Diffusion Models

Uniform-in-Time Weak Error Analysis for Stochastic Gradient Descent Algorithms Via Diffusion Approximation

On the generalization properties of diffusion models

Hitting the High-Dimensional Notes: An ODE for SGD learning dynamics on GLMs and multi-index models

Theory and Algorithms for Diffusion Processes on Riemannian Manifolds

Convergence of Score-Based Discrete Diffusion Models: A Discrete-Time Analysis

Unified Convergence Analysis for Score-Based Diffusion Models with Deterministic Samplers

On the Diffusion Approximation of Nonconvex Stochastic Gradient Descent

$O(d/T)$ Convergence Theory for Diffusion Probabilistic Models under Minimal Assumptions

Linear Convergence of Diffusion Models Under the Manifold Hypothesis

On Uniform-in-time Diffusion Approximation for Stochastic Gradient Descent.