Dynamical Regimes of Diffusion Models

Giulio Biroli,Tony Bonnaire,Valentin de Bortoli,Marc Mézard

2024-02-29

Abstract:Using statistical physics methods, we study generative diffusion models in the regime where the dimension of space and the number of data are large, and the score function has been trained optimally. Our analysis reveals three distinct dynamical regimes during the backward generative diffusion process. The generative dynamics, starting from pure noise, encounters first a 'speciation' transition where the gross structure of data is unraveled, through a mechanism similar to symmetry breaking in phase transitions. It is followed at later time by a 'collapse' transition where the trajectories of the dynamics become attracted to one of the memorized data points, through a mechanism which is similar to the condensation in a glass phase. For any dataset, the speciation time can be found from a spectral analysis of the correlation matrix, and the collapse time can be found from the estimation of an 'excess entropy' in the data. The dependence of the collapse time on the dimension and number of data provides a thorough characterization of the curse of dimensionality for diffusion models. Analytical solutions for simple models like high-dimensional Gaussian mixtures substantiate these findings and provide a theoretical framework, while extensions to more complex scenarios and numerical validations with real datasets confirm the theoretical predictions.

Machine Learning,Statistical Mechanics

What problem does this paper attempt to address?

This paper investigates the dynamic behavior of diffusion models in high-dimensional and large datasets. It reveals three different dynamic stages in the reverse generative process: a pure Brownian motion stage, a classification stage, and a collapse stage. The macroscopic structure of the data begins to emerge in the classification stage, similar to symmetry breaking in phase transitions. The dynamic trajectories in the collapse stage are attracted to the data points stored in memory, similar to condensation in the glass phase. The paper analyzes the timescales of these two transitions using statistical physics methods and finds that the size of the dataset needs to grow exponentially with the dimensionality to avoid collapse, i.e., overfitting. The study also provides theoretical support through a high-dimensional Gaussian mixture model and validates it on real-world datasets.

Dynamical Regimes of Diffusion Models

Generative diffusion in very large dimensions

Nonequilbrium physics of generative diffusion models

Lecture Notes in Probabilistic Diffusion Models

On the Generalization Properties of Diffusion Models

Manifolds, Random Matrices and Spectral Gaps: The geometric phases of generative diffusion

The statistical thermodynamics of generative diffusion models: Phase transitions, symmetry breaking and critical instability

Spontaneous Symmetry Breaking in Generative Diffusion Models

Spontaneous symmetry breaking in generative diffusion models *

Renormalizing Diffusion Models

On gauge freedom, conservativity and intrinsic dimensionality estimation in diffusion models

Losing dimensions: Geometric memorization in generative diffusion

How Much Is Enough? A Study on Diffusion Times in Score-Based Generative Models

Score-based generative diffusion with "active" correlated noise sources

Scaling Riemannian Diffusion Models

A Phase Transition in Diffusion Models Reveals the Hierarchical Nature of Data

Random walks in correlated diffusivity landscapes

Towards Faster Non-Asymptotic Convergence for Diffusion-Based Generative Models

Anomalous Fluctuations of Extremes in Many-Particle Diffusion

From memorization to generalization: a theoretical framework for diffusion-based generative models

An optimal control perspective on diffusion-based generative modeling