The statistical thermodynamics of generative diffusion models: Phase transitions, symmetry breaking and critical instability

Luca Ambrogioni

2024-06-20

Abstract:Generative diffusion models have achieved spectacular performance in many areas of machine learning and generative modeling. While the fundamental ideas behind these models come from non-equilibrium physics, variational inference and stochastic calculus, in this paper we show that many aspects of these models can be understood using the tools of equilibrium statistical mechanics. Using this reformulation, we show that generative diffusion models undergo second-order phase transitions corresponding to symmetry breaking phenomena. We show that these phase-transitions are always in a mean-field universality class, as they are the result of a self-consistency condition in the generative dynamics. We argue that the critical instability that arises from the phase transitions lies at the heart of their generative capabilities, which are characterized by a set of mean-field critical exponents. Finally, we show that the dynamic equation of the generative process can be interpreted as a stochastic adiabatic transformation that minimizes the free energy while keeping the system in thermal equilibrium.

Machine Learning

What problem does this paper attempt to address?

The paper investigates the statistical thermodynamic properties of the generative diffusion models in the field of machine learning and generative modeling, particularly the phenomena associated with phase transitions, symmetry breaking, and critical instability. The authors propose to reframe these models within the framework of equilibrium statistical physics and state that these models undergo second-order phase transitions, similar to spontaneous symmetry breaking. These transitions belong to the mean-field universality class and are caused by the self-consistency conditions of the generative dynamics. The paper also discusses that the critical instability in the phase transition is at the core of generative capabilities, characterized by a set of mean-field critical exponents. Additionally, the authors demonstrate that the dynamics equation of the generative process can be interpreted as a stochastic adiabatic transformation that minimizes the free energy while keeping the system in a thermally balanced state. Finally, the paper proposes a generative bath model for a multi-site system as an extension of mean-field theory.

The statistical thermodynamics of generative diffusion models: Phase transitions, symmetry breaking and critical instability

Nonequilbrium physics of generative diffusion models

Thermodynamic Fidelity of Generative Models for Ising System

Thermodynamics of Markov Processes with Non-extensive Entropy and Free Energy

Dynamical Regimes of Diffusion Models

Manifolds, Random Matrices and Spectral Gaps: The geometric phases of generative diffusion

Spontaneous symmetry breaking in generative diffusion models *

Phase transitions on heterogeneous random graphs: some case studies

Generative diffusion in very large dimensions

Spontaneous Symmetry Breaking in Generative Diffusion Models

A statistical mechanics framework for constructing non-equilibrium thermodynamic models

Learning nonequilibrium statistical mechanics and dynamical phase transitions

Inferring phase transitions and critical exponents from limited observations with Thermodynamic Maps

Dynamical phase transitions in the non-reciprocal Ising model

Microscopic statistical basis of classical Thermodynamics of finite systems

Statistical Mechanics Approaches for Studying Temperature and Rate Effects in Multistable Systems

A thermodynamical model for concurrent diffusive and displacive phase transitions

Heat can flow from cold to hot in Microcanonical Thermodynamics of finite systems and the microscopic origin of phase transitions

Generalized time-reversal symmetry and effective theories for nonequilibrium matter

Phase Transition and Thermodynamic Stability in an Entropy-driven Universe

Stochastic Thermodynamics Across Scales: Emergent Inter-attractoral Discrete Markov Jump Process and Its Underlying Continuous Diffusion