Abstract:The number of sampling methods could be daunting for a practitioner looking to cast powerful machine learning methods to their specific problem. This paper takes a theoretical stance to review and organize many sampling approaches in the ``generative modeling'' setting, where one wants to generate new data that are similar to some training examples. By revealing links between existing methods, it might prove useful to overcome some of the current challenges in sampling with diffusion models, such as long inference time due to diffusion simulation, or the lack of diversity in generated samples.
What problem does this paper attempt to address?
This paper explores how to effectively perform sampling in the setting of generative modeling in machine learning, particularly addressing the issues of long inference time and lack of sample diversity in diffusion models. The authors provide theoretical analysis and review various sampling methods, aiming to reveal the connections between existing methods in order to overcome these challenges.
The core of the paper is to learn the mapping between distributions through the exploration and organization of deterministic mapping learning strategies, in order to achieve fast inference algorithms. The authors distinguish three different perspectives based on test functions, density likelihood, and probability flow. The paper mentions integral probability metrics, maximum mean discrepancy (MMD), generative adversarial networks (GANs), diffusion models, and flow models, discussing their advantages and disadvantages.
Statistical matching aims to ensure that the generated distribution matches the target distribution by comparing different metrics on test functions. The paper proposes two forms of statistical matching, maximum and average, as well as methods using feature function matching. The authors also study density level methods, such as learning distributions by maximizing the likelihood of probability models, and introduce the working principles of variational autoencoders (VAEs).
In the third part, the authors focus on constructing mappings from flows, using stochastic differential equations (such as Ornstein-Uhlenbeck process) to describe continuous transformations from base distributions to target distributions, and discuss how to deduce deterministic mappings from simulated processes.
Finally, the paper suggests that although diffusion models perform well in practice, they have slow inference speed and require expensive simulations. The authors propose directly learning the forward mapping instead of learning and integrating flows, which may help improve the optimization process, particularly for GANs training.
Overall, this paper attempts to address how to unify and optimize various sampling methods through a theoretical framework, to improve the efficiency and sample diversity of diffusion models, and provide more effective tools for generative modeling.