Abstract:Bayesian approaches have been successfully integrated into training deep neural networks. One popular family is stochastic gradient Markov chain Monte Carlo methods (SG-MCMC), which have gained increasing interest due to their scalability to handle large datasets and the ability to avoid overfitting. Although standard SG-MCMC methods have shown great performance in a variety of problems, they may be inefficient when the random variables in the target posterior densities have scale differences or are highly correlated. In this work, we present an adaptive Hessian approximated stochastic gradient MCMC method to incorporate local geometric information while sampling from the posterior. The idea is to apply stochastic approximation to sequentially update a preconditioning matrix at each iteration. The preconditioner possesses second-order information and can guide the random walk of a sampler efficiently. Instead of computing and saving the full Hessian of the log posterior, we use limited memory of the sample and their stochastic gradients to approximate the inverse Hessian-vector multiplication in the updating formula. Moreover, by smoothly optimizing the preconditioning matrix, our proposed algorithm can asymptotically converge to the target distribution with a controllable bias under mild conditions. To reduce the training and testing computational burden, we adopt a magnitude-based weight pruning method to enforce the sparsity of the network. Our method is user-friendly and is scalable to standard SG-MCMC updating rules by implementing an additional preconditioner. The sparse approximation of inverse Hessian alleviates storage and computational complexities for large dimensional models. The bias introduced by stochastic approximation is controllable and can be analyzed theoretically. Numerical experiments are performed on several problems.

Gradient-flow adaptive importance sampling for Bayesian leave one out cross-validation with application to sigmoidal classification models

Low-Cost Lipschitz-Independent Adaptive Importance Sampling of Stochastic Gradients.

Multiple importance sampling for stochastic gradient estimation

Gradient-based Sampling: An Adaptive Importance Sampling for Least-squares

Importance Sampled Stochastic Optimization for Variational Inference

Importance sampling for rare-event gradient estimation

Weighted Leave-One-Out Cross Validation

Bayesian leave-one-out cross-validation for large data

Variational Learning of Gaussian Process Latent Variable Models through Stochastic Gradient Annealed Importance Sampling

Gradient Importance Sampling

Stochastic Gradient Hamiltonian Monte Carlo with Variance Reduction for Bayesian Inference.

Stochastic optimization with importance sampling for regularized loss minimization

Non-convex Bayesian Learning via Stochastic Gradient Markov Chain Monte Carlo

Sampling from Gaussian Process Posteriors using Stochastic Gradient Descent

An adaptive Hessian approximated stochastic gradient MCMC method

Unbiased Kinetic Langevin Monte Carlo with Inexact Gradients

Optimal sampling for stochastic and natural gradient descent

Scalable Stochastic Gradient Riemannian Langevin Dynamics in Non-Diagonal Metrics

Nonlinear Monte Carlo Method for Imbalanced Data Learning

Deep importance sampling based on regression for model inversion and emulation

On the ISS Property of the Gradient Flow for Single Hidden-Layer Neural Networks with Linear Activations