Continuous-Time Flows for Efficient Inference and Density Estimation

Changyou Chen,Chunyuan Li,Liqun Chen,Wenlin Wang,Yunchen Pu,Lawrence Carin

DOI: https://doi.org/10.48550/arXiv.1709.01179

2018-08-01

Abstract:Two fundamental problems in unsupervised learning are efficient inference for latent-variable models and robust density estimation based on large amounts of unlabeled data. Algorithms for the two tasks, such as normalizing flows and generative adversarial networks (GANs), are often developed independently. In this paper, we propose the concept of {\em continuous-time flows} (CTFs), a family of diffusion-based methods that are able to asymptotically approach a target distribution. Distinct from normalizing flows and GANs, CTFs can be adopted to achieve the above two goals in one framework, with theoretical guarantees. Our framework includes distilling knowledge from a CTF for efficient inference, and learning an explicit energy-based distribution with CTFs for density estimation. Both tasks rely on a new technique for distribution matching within amortized learning. Experiments on various tasks demonstrate promising performance of the proposed CTF framework, compared to related techniques.

Machine Learning

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to achieve efficient inference and robust probability density estimation in unsupervised learning. Specifically, the authors propose the concept of Continuous - Time Flows (CTFs), a diffusion - based method that can asymptotically approach the target distribution. Unlike traditional Normalizing Flows (NFs) and Generative Adversarial Networks (GANs), CTFs can achieve both of the above - mentioned goals within one framework and have theoretical guarantees.

### Main Contributions
1. **Unified Framework**: CTFs provide a unified framework that can be used for both efficient inference and probability density estimation simultaneously.
2. **Theoretical Guarantee**: CTFs can asymptotically approach the target distribution, providing a theoretical guarantee.
3. **New Technique**: A new distribution - matching technique is introduced to optimize the model during the amortized learning process.

### Specific Problems
- **Efficient Inference**: In Bayesian models, the goal is to learn a tractable latent variable distribution from a given unnormalized distribution, making it as close as possible to the posterior distribution.
- **Probability Density Estimation**: Learn the unknown data distribution based solely on sample data.

### Method Overview
- **Continuous - Time Flows (CTFs)**: Through a series of continuous - time transformations, the distribution of random variables evolves from a simple distribution to a complex one.
- **Amortized Learning**: Through the distribution - matching technique, the knowledge of CTFs is gradually distilled into another neural network, thereby achieving efficient inference and density estimation.
- **Explicit and Implicit Methods**: CTFs can be used to learn the unknown data distribution in an explicit form and can also be used to generate samples, thus combining the advantages of explicit and implicit methods.

### Experimental Results
The paper demonstrates the superior performance of the proposed CTF framework over existing methods through experiments on synthetic data and real - world datasets.

### Formulas
- **Langevin Dynamics**:
\[
dZ_t = F(Z_t)dt+V(Z_t)dW
\]
where \(F(Z_t)\) is the drift term, \(V(Z_t)\) is the diffusion term, and \(W\) is a standard \(L\)-dimensional Brownian motion.

- **Fokker - Planck Equation**:
\[
\frac{\partial\rho_t}{\partial t}=-\nabla_z\cdot(\rho_tF(Z_t)+\nabla_z\cdot(\rho_tV(Z_t)V^{\top}(Z_t)))
\]

- **Negative ELBO**:
\[
F(x)=\mathbb{E}_{q_{\phi}(z_0|x)}\mathbb{E}_{\rho_T}\left[\log\rho_T-\log p_{\theta}(x,Z_T)+\log\left|\det\frac{\partial Z_T}{\partial z_0}\right|\right]
\]

- **Wasserstein Distance**:
\[
W_2^2(\mu_1,\mu_2)=\inf_{p\in P(\mu_1,\mu_2)}\int\|x - y\|^2p(dx,dy)
\]

### Conclusion
The paper proposes a new continuous - time flow method that can simultaneously achieve efficient inference and robust probability density estimation in unsupervised learning, with theoretical guarantees and superior performance in practical applications.