Abstract:Diffusion models are generative models that have recently demonstrated impressive performances in terms of sampling quality and density estimation in high dimensions. They rely on a forward continuous diffusion process and a backward continuous denoising process, which can be described by a time-dependent vector field and is used as a generative model. In the original formulation of the diffusion model, this vector field is assumed to be the score function (i.e. it is the gradient of the log-probability at a given time in the diffusion process). Curiously, on the practical side, most studies on diffusion models implement this vector field as a neural network function and do not constrain it be the gradient of some energy function (that is, most studies do not constrain the vector field to be conservative). Even though some studies investigated empirically whether such a constraint will lead to a performance gain, they lead to contradicting results and failed to provide analytical results. Here, we provide three analytical results regarding the extent of the modeling freedom of this vector field. {Firstly, we propose a novel decomposition of vector fields into a conservative component and an orthogonal component which satisfies a given (gauge) freedom. Secondly, from this orthogonal decomposition, we show that exact density estimation and exact sampling is achieved when the conservative component is exactly equals to the true score and therefore conservativity is neither necessary nor sufficient to obtain exact density estimation and exact sampling. Finally, we show that when it comes to inferring local information of the data manifold, constraining the vector field to be conservative is desirable.

Scalable reversible generative models with free-form continuous dynamics

Ffjord: Free-form continuous dynamics for scalable reversible generative models

From pixels to planning: scale-free active inference

Free-form Flows: Make Any Architecture a Normalizing Flow

Adaptive Density Estimation for Generative Models

Trans-Dimensional Generative Modeling via Jump Diffusion Models

Variational Autoencoders for Efficient Simulation-Based Inference

Continual learning with invertible generative models

Stable generative modeling using Schrödinger bridges

Generative modeling of time-dependent densities via optimal transport and projection pursuit

Generative models for physicists

On gauge freedom, conservativity and intrinsic dimensionality estimation in diffusion models

Generative Assignment Flows for Representing and Learning Joint Distributions of Discrete Data

Generative learning for nonlinear dynamics

Monge-Ampère Flow for Generative Modeling

Convergence of flow-based generative models via proximal gradient descent in Wasserstein space

Stochastic interpolants with data-dependent couplings

AmbientFlow: Invertible generative models from incomplete, noisy measurements

Continuously indexed graphical models