Abstract:By learning the gradient of smoothed data distributions, diffusion models can iteratively generate samples from complex distributions. The learned score function enables their generalization capabilities, but how the learned score relates to the score of the underlying data manifold remains largely unclear. Here, we aim to elucidate this relationship by comparing learned neural scores to the scores of two kinds of analytically tractable distributions: Gaussians and Gaussian mixtures. The simplicity of the Gaussian model makes it theoretically attractive, and we show that it admits a closed-form solution and predicts many qualitative aspects of sample generation dynamics. We claim that the learned neural score is dominated by its linear (Gaussian) approximation for moderate to high noise scales, and supply both theoretical and empirical arguments to support this claim. Moreover, the Gaussian approximation empirically works for a larger range of noise scales than naive theory suggests it should, and is preferentially learned early in training. At smaller noise scales, we observe that learned scores are better described by a coarse-grained (Gaussian mixture) approximation of training data than by the score of the training distribution, a finding consistent with generalization. Our findings enable us to precisely predict the initial phase of trained models' sampling trajectories through their Gaussian approximations. We show that this allows the skipping of the first 15-30% of sampling steps while maintaining high sample quality (with a near state-of-the-art FID score of 1.93 on CIFAR-10 unconditional generation). This forms the foundation of a novel hybrid sampling method, termed analytical teleportation, which can seamlessly integrate with and accelerate existing samplers, including DPM-Solver-v3 and UniPC. Our findings suggest ways to improve the design and training of diffusion models.

Multilevel Diffusion: Infinite Dimensional Score-Based Diffusion Models for Image Generation

Conditional score-based diffusion models for Bayesian inference in infinite dimensions

Where to Diffuse, How to Diffuse, and How to Get Back: Automated Learning for Multivariate Diffusions

Adapting to Unknown Low-Dimensional Structures in Score-Based Diffusion Models

Score Approximation, Estimation and Distribution Recovery of Diffusion Models on Low-Dimensional Data

Continuous-Time Functional Diffusion Processes

SDDM: Score-Decomposed Diffusion Models on Manifolds for Unpaired Image-to-Image Translation

What's the score? Automated Denoising Score Matching for Nonlinear Diffusions

Non-Uniform Diffusion Models

Score-based Diffusion Models for Bayesian Image Reconstruction

The Unreasonable Effectiveness of Gaussian Score Approximation for Diffusion Models and its Applications

Diffusion models for Gaussian distributions: Exact solutions and Wasserstein errors

Generative Fractional Diffusion Models

A Complete Recipe for Diffusion Generative Models

Convergence of Score-Based Discrete Diffusion Models: A Discrete-Time Analysis

A Flexible Diffusion Model

Diffusion Models Without Attention

Diffusion Schrödinger Bridge with Applications to Score-Based Generative Modeling

How Much Is Enough? A Study on Diffusion Times in Score-Based Generative Models

Simulating infinite-dimensional nonlinear diffusion bridges

Blackout Diffusion: Generative Diffusion Models in Discrete-State Spaces