Abstract:Bayesian approaches have been successfully integrated into training deep neural networks. One popular family is stochastic gradient Markov chain Monte Carlo methods (SG-MCMC), which have gained increasing interest due to their scalability to handle large datasets and the ability to avoid overfitting. Although standard SG-MCMC methods have shown great performance in a variety of problems, they may be inefficient when the random variables in the target posterior densities have scale differences or are highly correlated. In this work, we present an adaptive Hessian approximated stochastic gradient MCMC method to incorporate local geometric information while sampling from the posterior. The idea is to apply stochastic approximation to sequentially update a preconditioning matrix at each iteration. The preconditioner possesses second-order information and can guide the random walk of a sampler efficiently. Instead of computing and saving the full Hessian of the log posterior, we use limited memory of the sample and their stochastic gradients to approximate the inverse Hessian-vector multiplication in the updating formula. Moreover, by smoothly optimizing the preconditioning matrix, our proposed algorithm can asymptotically converge to the target distribution with a controllable bias under mild conditions. To reduce the training and testing computational burden, we adopt a magnitude-based weight pruning method to enforce the sparsity of the network. Our method is user-friendly and is scalable to standard SG-MCMC updating rules by implementing an additional preconditioner. The sparse approximation of inverse Hessian alleviates storage and computational complexities for large dimensional models. The bias introduced by stochastic approximation is controllable and can be analyzed theoretically. Numerical experiments are performed on several problems.

A globally convergent approximate Newton method for non-convex sparse learning

Convergence of Projected Subgradient Method with Sparse or Low-Rank Constraints

Quadratic Approximation Greedy Pursuit for Cardinality-Constrained Sparse Learning

Distributed Inexact Newton-Type Pursuit For Non-Convex Sparse Learning

A Semismooth Newton Algorithm for High-Dimensional Nonconvex Sparse Learning

A Non-convex Approach for Sparse Recovery with Convergence Guarantee

The Convergence Guarantees of a Non-Convex Approach for Sparse Recovery.

A Multilevel Low-Rank Newton Method with Super-linear Convergence Rate and its Application to Non-convex Problems

Globally Convergent Newton Methods for Ill-conditioned Generalized Self-concordant Losses

Approximate Newton Methods and Their Local Convergence.

Efficient Projected Gradient Methods for Cardinality Constrained Optimization

A globally convergent proximal Newton-type method in nonsmooth convex optimization

Online Learning Guided Curvature Approximation: A Quasi-Newton Method with Global Non-Asymptotic Superlinear Convergence

Optimal computational and statistical rates of convergence for sparse nonconvex learning problems

A Unifying Framework for Convergence Analysis of Approximate Newton Methods.

Projected Neural Network for a Class of Sparse Regression with Cardinality Penalty

SGN: Sparse Gauss-Newton for Accelerated Sensitivity Analysis

A Smoothing Proximal Gradient Algorithm for Nonsmooth Convex Regression with Cardinality Penalty.

SPAN: A Stochastic Projected Approximate Newton Method

Subspace Newton method for sparse group optimization problem

An adaptive Hessian approximated stochastic gradient MCMC method