Diffusion-Model-Assisted Supervised Learning of Generative Models for Density Estimation

Yanfang Liu,Minglei Yang,Zezhong Zhang,Feng Bao,Yanzhao Cao,Guannan Zhang
2023-10-23
Abstract:We present a supervised learning framework of training generative models for density estimation. Generative models, including generative adversarial networks, normalizing flows, variational auto-encoders, are usually considered as unsupervised learning models, because labeled data are usually unavailable for training. Despite the success of the generative models, there are several issues with the unsupervised training, e.g., requirement of reversible architectures, vanishing gradients, and training instability. To enable supervised learning in generative models, we utilize the score-based diffusion model to generate labeled data. Unlike existing diffusion models that train neural networks to learn the score function, we develop a training-free score estimation method. This approach uses mini-batch-based Monte Carlo estimators to directly approximate the score function at any spatial-temporal location in solving an ordinary differential equation (ODE), corresponding to the reverse-time stochastic differential equation (SDE). This approach can offer both high accuracy and substantial time savings in neural network training. Once the labeled data are generated, we can train a simple fully connected neural network to learn the generative model in the supervised manner. Compared with existing normalizing flow models, our method does not require to use reversible neural networks and avoids the computation of the Jacobian matrix. Compared with existing diffusion models, our method does not need to solve the reverse-time SDE to generate new samples. As a result, the sampling efficiency is significantly improved. We demonstrate the performance of our method by applying it to a set of 2D datasets as well as real data from the UCI repository.
Machine Learning,Numerical Analysis
What problem does this paper attempt to address?
The problem that this paper attempts to solve is to achieve supervised learning for density estimation tasks in generative models. Generally, generative models such as Generative Adversarial Networks (GANs), Normalizing Flows, and Variational Auto - Encoders (VAEs) are considered unsupervised learning models because there is often no labeled data available when training these models. This has led to some problems during the training process, such as the need for invertible architectures, vanishing gradients, and training instability. To solve these problems, the paper proposes a new framework. By using score - based diffusion models to generate labeled data, generative models can be trained in a supervised learning manner. The core of this method lies in developing a score estimation method without training, which directly approximates the score function in ordinary differential equations (ODEs) using a mini - batch - based Monte Carlo estimator. This can not only provide high - precision estimates but also significantly save the time cost of neural network training. Once the labeled data is generated, a simple fully - connected neural network can be used to learn the generative model in a supervised learning way. Compared with existing Normalizing Flow models, this method does not need to use invertible neural networks, avoiding the calculation of the Jacobian matrix; compared with existing diffusion models, this method does not need to solve the backward - time stochastic differential equation (SDE) to generate new samples, thus significantly improving the sampling efficiency. The paper demonstrates its performance by applying this method to a series of 2D datasets and real data in the UCI repository.