Abstract:We present a supervised learning framework of training generative models for density estimation. Generative models, including generative adversarial networks, normalizing flows, variational auto-encoders, are usually considered as unsupervised learning models, because labeled data are usually unavailable for training. Despite the success of the generative models, there are several issues with the unsupervised training, e.g., requirement of reversible architectures, vanishing gradients, and training instability. To enable supervised learning in generative models, we utilize the score-based diffusion model to generate labeled data. Unlike existing diffusion models that train neural networks to learn the score function, we develop a training-free score estimation method. This approach uses mini-batch-based Monte Carlo estimators to directly approximate the score function at any spatial-temporal location in solving an ordinary differential equation (ODE), corresponding to the reverse-time stochastic differential equation (SDE). This approach can offer both high accuracy and substantial time savings in neural network training. Once the labeled data are generated, we can train a simple fully connected neural network to learn the generative model in the supervised manner. Compared with existing normalizing flow models, our method does not require to use reversible neural networks and avoids the computation of the Jacobian matrix. Compared with existing diffusion models, our method does not need to solve the reverse-time SDE to generate new samples. As a result, the sampling efficiency is significantly improved. We demonstrate the performance of our method by applying it to a set of 2D datasets as well as real data from the UCI repository.

Roundtrip: A Deep Generative Neural Density Estimator

Density Estimation Using Deep Generative Neural Networks.

Depth Generation Network: Estimating Real World Depth From Stereo And Depth Images

Neural-g: A Deep Learning Framework for Mixing Density Estimation

Deep data density estimation through Donsker-Varadhan representation

Data-driven deep density estimation

Generative modeling of time-dependent densities via optimal transport and projection pursuit

On the capacity of deep generative networks for approximating distributions

Rethinking LiDAR Domain Generalization: Single Source as Multiple Density Domains

Density-Regression: Efficient and Distance-Aware Deep Regressor for Uncertainty Estimation under Distribution Shifts

Generative Adversarial Nets for Robust Scatter Estimation: A Proper Scoring Rule Perspective

Fast Deep Mixtures of Gaussian Process Experts

A supervised deep learning method for nonparametric density estimation

Generalized Gumbel-Softmax Gradient Estimator for Generic Discrete Random Variables

A Deep Generative Approach to Conditional Sampling

Density regression and uncertainty quantification with Bayesian deep noise neural networks

Diffusion Density Estimators

Deep-plane sweep generative adversarial network for consistent multi-view depth estimation

Diffusion-Model-Assisted Supervised Learning of Generative Models for Density Estimation

Bias and Generalization in Deep Generative Models: An Empirical Study