Abstract:Sampling from high-dimensional, multi-modal distributions remains a fundamental challenge across domains such as statistical Bayesian inference and physics-based machine learning. In this paper, we propose Annealing Flow (AF), a continuous normalizing flow-based approach designed to sample from high-dimensional and multi-modal distributions. The key idea is to learn a continuous normalizing flow-based transport map, guided by annealing, to transition samples from an easy-to-sample distribution to the target distribution, facilitating effective exploration of modes in high-dimensional spaces. Unlike many existing methods, AF training does not rely on samples from the target distribution. AF ensures effective and balanced mode exploration, achieves linear complexity in sample size and dimensions, and circumvents inefficient mixing times. We demonstrate the superior performance of AF compared to state-of-the-art methods through extensive experiments on various challenging distributions and real-world datasets, particularly in high-dimensional and multi-modal settings. We also highlight the potential of AF for sampling the least favorable distributions.
What problem does this paper attempt to address?
### What problem does this paper attempt to solve?
This paper aims to solve the fundamental challenge of sampling from high - dimensional, multimodal distributions. Specifically, the authors propose the **Annealing Flow (AF)** method, a sampling technique based on Continuous Normalizing Flow, for efficiently generating samples from high - dimensional and multimodal distributions.
#### Key issues:
1. **Sampling difficulty of high - dimensional and multimodal distributions**: In fields such as statistical Bayesian inference and physics - based machine learning, effectively sampling from complex and multimodal high - dimensional distributions has always been a difficult problem.
2. **Limitations of existing methods**:
- **MCMC methods**: Such as Metropolis - Hastings (MH) and Hamiltonian Monte Carlo (HMC), the mixing time grows exponentially in high - dimensional space, resulting in inefficiency and being prone to getting trapped in local modes.
- **Particle optimization methods**: Such as Stein Variational Gradient Descent (SVGD), although it avoids the warm - up period and reduces sample correlation, the computational complexity grows polynomially with the sample size and dimension and is sensitive to hyperparameters.
- **Neural network (NN) - based sampling algorithms**: Despite taking advantage of the expressive power of neural networks, they still inherit the problems of slow mixing and unbalanced mode exploration in high - dimensional cases.
#### Solutions:
- **Annealing Flow (AF)**: By learning a continuous normalizing flow mapping, gradually transform the initially easy - to - sample distribution into the target distribution. The key idea of AF is to combine the annealing process, allowing samples to smoothly transition from a simple distribution to the complex target distribution.
- **Does not rely on samples from the target distribution**: Unlike many existing methods, AF training does not require samples from the target distribution, thus avoiding the problems of inefficient mixing time and unbalanced mode exploration.
- **Linear complexity**: AF achieves linear complexity in sample size and dimension, significantly improving sampling efficiency.
- **Balanced mode exploration**: Ensure effective exploration of all modes in high - dimensional space, especially performing well when dealing with multimodal distributions.
### Summary
This paper provides a novel and efficient method for sampling from high - dimensional, multimodal distributions by introducing Annealing Flow, solving the limitations of existing methods in high - dimensional space, especially outstanding in mode exploration and sampling efficiency.