Abstract:Diffusion-based generative models use stochastic differential equations (SDEs) and their equivalent ordinary differential equations (ODEs) to establish a smooth connection between a complex data distribution and a tractable prior distribution. In this paper, we identify several intriguing trajectory properties in the ODE-based sampling process of diffusion models. We characterize an implicit denoising trajectory and discuss its vital role in forming the coupled sampling trajectory with a strong shape regularity, regardless of the generated content. We also describe a dynamic programming-based scheme to make the time schedule in sampling better fit the underlying trajectory structure. This simple strategy requires minimal modification to any given ODE-based numerical solvers and incurs negligible computational cost, while delivering superior performance in image generation, especially in $5\sim 10$ function evaluations.

What problem does this paper attempt to address?

The paper explores the regularity of trajectory in diffusion sampling based on differential equations. In diffusion generative models, these models establish a smooth connection between complex data distributions and tractable prior distributions through stochastic differential equations (SDEs) and their equivalent ordinary differential equations (ODEs). It is found that there are interesting trajectory characteristics in the sampling process based on ODEs, namely, there exists an implicit denoising trajectory that plays a key role in forming regular sampling trajectories, regardless of the generated content. Additionally, the paper describes a dynamic programming-based method to optimize the sampling time allocation to better adapt to the underlying trajectory structure. This method requires minimal modifications to existing ODE numerical solvers but significantly improves image generation performance, especially in few function evaluations. Specifically, the paper observes that the sampling trajectories tend to exhibit a linear-nonlinear-linear "boomerang" shape, and each sampling trajectory moves almost along straight lines between the starting point and ending point. This pattern consistently appears in different initial random samples or corresponding real data samples. The authors propose that this regularity reflects some underlying geometric structures of the sampling trajectories, and they model the data distribution through variable-width kernel density estimation (KDE), revealing the closed form of the denoising trajectory. Although this KDE-based solution is not applicable to practical sampling trajectory solving, it theoretically converges to the optimal solution based on the real data distribution. Utilizing the regularity of sampling trajectory shape, the paper proposes an accelerated sampling method based on dynamic programming to determine the optimal sampling time step. Experimental results show that accelerated sampling based on trajectory regularity significantly improves the performance of diffusion-based generative models in few function evaluations.

On the Trajectory Regularity of ODE-based Diffusion Sampling

A Geometric Perspective on Diffusion Models

New algorithms for sampling and diffusion models

Exploring the Optimal Choice for Generative Processes in Diffusion Models: Ordinary vs Stochastic Differential Equations

Fast ODE-based Sampling for Diffusion Models in Around 5 Steps

Geometric Trajectory Diffusion Models

Learning to Approximate Particle Smoothing Trajectories via Diffusion Generative Models

Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions

Distilling ODE Solvers of Diffusion Models into Smaller Steps

Adaptive Non-Uniform Timestep Sampling for Diffusion Model Training

Diffusion Generative Flow Samplers: Improving learning signals through partial trajectory optimization

Model-Based Diffusion for Trajectory Optimization

Towards Faster Non-Asymptotic Convergence for Diffusion-Based Generative Models

Accelerating Diffusion Sampling with Optimized Time Steps

Learning Quantized Adaptive Conditions for Diffusion Models

A Sharp Convergence Theory for The Probability Flow ODEs of Diffusion Models

On Accelerating Diffusion-Based Sampling Process via Improved Integration Approximation

Consistency Trajectory Models: Learning Probability Flow ODE Trajectory of Diffusion

A Unified Sampling Framework for Solver Searching of Diffusion Probabilistic Models

Deterministic Gibbs Sampling via Ordinary Differential Equations

Stochastic Runge-Kutta Methods: Provable Acceleration of Diffusion Models