Abstract:Diffusions are a successful technique to sample from high-dimensional distributions can be either explicitly given or learnt from a collection of samples. They implement a diffusion process whose endpoint is a sample from the target distribution and whose drift is typically represented as a neural network. Stochastic localization is a successful technique to prove mixing of Markov Chains and other functional inequalities in high dimension. An algorithmic version of stochastic localization was introduced in [EAMS2022], to obtain an algorithm that samples from certain statistical mechanics models. This notes have three objectives: (i) Generalize the construction [EAMS2022] to other stochastic localization processes; (ii) Clarify the connection between diffusions and stochastic localization. In particular we show that standard denoising diffusions are stochastic localizations but other examples that are naturally suggested by the proposed viewpoint; (iii) Describe some insights that follow from this viewpoint.

What problem does this paper attempt to address?

The paper primarily explores methods of sampling from given distributions in high-dimensional spaces, with a focus on diffusion processes and stochastic localization techniques. The authors begin by introducing diffusion processes as a successful sampling technique, capable of sampling from high-dimensional distributions that are either explicitly given or learned from a sample set. Diffusion processes operate by implementing a diffusion whose endpoint is sampled from the target distribution, and its drift is often represented by neural networks. Subsequently, the paper discusses stochastic localization, a technique that has proven successful in proving Markov chain mixing and other functional inequalities in high dimensions. The three main objectives of the paper are: (i) to extend previous research findings to other stochastic localization processes; (ii) to clarify the connection between diffusion processes and stochastic localization, particularly demonstrating that standard denoising diffusions are actually a form of stochastic localization; (iii) to describe some insights derived from this perspective. The paper also provides a detailed comparison between the two methods of diffusion processes and stochastic localization, noting that they are inherently related but offer different perspectives and methods that may simplify the learning task. By choosing the appropriate stochastic localization process, the learning task in standard denoising diffusion can be simplified. Moreover, the paper outlines relevant literature and presents several specific instances, showing how to use the current viewpoint to address problems that arise in standard denoising diffusion. Finally, the paper illustrates through several concrete examples how to construct and use different observation processes for sampling, including isotropic Gaussian processes, anisotropic Gaussian processes, erasure processes, binary symmetric processes, linear observation processes, information percolation processes, Poisson observation processes, and half-space processes. These observation processes provide new avenues for sampling from complex distributions and can be combined to suit specific sampling needs.

Sampling, Diffusions, and Stochastic Localization

Stochastic Localization via Iterative Posterior Sampling

Denoising Diffusion Samplers

Posterior Sampling in High Dimension via Diffusion Processes

On diffusion models for amortized inference: Benchmarking and improving stochastic control and sampling

New algorithms for sampling and diffusion models

Sequential Controlled Langevin Diffusions

Improved sampling via learned diffusions

Sampling and estimation on manifolds using the Langevin diffusion

A variational approach to sampling in diffusion processes

Sampling via Gradient Flows in the Space of Probability Measures

Implicit Diffusion: Efficient Optimization through Stochastic Sampling

A Geometric Perspective on Diffusion Models

Learned Reference-based Diffusion Sampling for multi-modal distributions

A Practical Diffusion Path for Sampling

Sampling from Mean-Field Gibbs Measures via Diffusion Processes

To smooth a cloud or to pin it down: Guarantees and Insights on Score Matching in Denoising Diffusion Models

Sampling with flows, diffusion and autoregressive neural networks: A spin-glass perspective

Schr{\"o}dinger-F{\"o}llmer Sampler: Sampling without Ergodicity

Faster Sampling without Isoperimetry via Diffusion-based Monte Carlo

From Denoising Diffusions to Denoising Markov Models