Abstract:Bayesian approaches have been successfully integrated into training deep neural networks. One popular family is stochastic gradient Markov chain Monte Carlo methods (SG-MCMC), which have gained increasing interest due to their scalability to handle large datasets and the ability to avoid overfitting. Although standard SG-MCMC methods have shown great performance in a variety of problems, they may be inefficient when the random variables in the target posterior densities have scale differences or are highly correlated. In this work, we present an adaptive Hessian approximated stochastic gradient MCMC method to incorporate local geometric information while sampling from the posterior. The idea is to apply stochastic approximation to sequentially update a preconditioning matrix at each iteration. The preconditioner possesses second-order information and can guide the random walk of a sampler efficiently. Instead of computing and saving the full Hessian of the log posterior, we use limited memory of the sample and their stochastic gradients to approximate the inverse Hessian-vector multiplication in the updating formula. Moreover, by smoothly optimizing the preconditioning matrix, our proposed algorithm can asymptotically converge to the target distribution with a controllable bias under mild conditions. To reduce the training and testing computational burden, we adopt a magnitude-based weight pruning method to enforce the sparsity of the network. Our method is user-friendly and is scalable to standard SG-MCMC updating rules by implementing an additional preconditioner. The sparse approximation of inverse Hessian alleviates storage and computational complexities for large dimensional models. The bias introduced by stochastic approximation is controllable and can be analyzed theoretically. Numerical experiments are performed on several problems.

Stochastic Hessian Fittings with Lie Groups

Curvature-Informed SGD via General Purpose Lie-Group Preconditioners

Riemannian optimization with a preconditioning scheme on the generalized Stiefel manifold

Stochastic Gradient Methods with Preconditioned Updates

An efficient Hessian based algorithm for solving large-scale sparse group Lasso problems

Optimization without Retraction on the Random Generalized Stiefel Manifold

Stochastic proximal gradient methods for nonconvex problems in Hilbert spaces

Riemannian Stochastic Proximal Gradient Methods for Nonsmooth Optimization over the Stiefel Manifold

Detecting negative eigenvalues of exact and approximate Hessian matrices in optimization

Simpler Grassmannian optimization

Riemannian optimization on the symplectic Stiefel manifold using second-order information

A Geometric Perspective on Fusing Gaussian Distributions on Lie Groups

The Stochastic Steepest Descent Method for Robust Optimization in Banach Spaces

Nonlinear Eigen-approach ADMM for Sparse Optimization on Stiefel Manifold

Spectral Preconditioning for Gradient Methods on Graded Non-convex Functions

Useful Compact Representations for Data-Fitting

Orthogonal Directions Constrained Gradient Method: from non-linear equality constraints to Stiefel manifold

Approximating Hessian matrices using Bayesian inference: a new approach for quasi-Newton methods in stochastic optimization

An adaptive Hessian approximated stochastic gradient MCMC method

Algebraic multigrid preconditioning of the Hessian in PDE-constrained optimization