Abstract:Bayesian approaches have been successfully integrated into training deep neural networks. One popular family is stochastic gradient Markov chain Monte Carlo methods (SG-MCMC), which have gained increasing interest due to their scalability to handle large datasets and the ability to avoid overfitting. Although standard SG-MCMC methods have shown great performance in a variety of problems, they may be inefficient when the random variables in the target posterior densities have scale differences or are highly correlated. In this work, we present an adaptive Hessian approximated stochastic gradient MCMC method to incorporate local geometric information while sampling from the posterior. The idea is to apply stochastic approximation to sequentially update a preconditioning matrix at each iteration. The preconditioner possesses second-order information and can guide the random walk of a sampler efficiently. Instead of computing and saving the full Hessian of the log posterior, we use limited memory of the sample and their stochastic gradients to approximate the inverse Hessian-vector multiplication in the updating formula. Moreover, by smoothly optimizing the preconditioning matrix, our proposed algorithm can asymptotically converge to the target distribution with a controllable bias under mild conditions. To reduce the training and testing computational burden, we adopt a magnitude-based weight pruning method to enforce the sparsity of the network. Our method is user-friendly and is scalable to standard SG-MCMC updating rules by implementing an additional preconditioner. The sparse approximation of inverse Hessian alleviates storage and computational complexities for large dimensional models. The bias introduced by stochastic approximation is controllable and can be analyzed theoretically. Numerical experiments are performed on several problems.

Gradient-enhanced High Dimensional Model Representation Via Bayesian Inference

A PC-Kriging-HDMR integrated with an adaptive sequential sampling strategy for high-dimensional approximate modeling

Surrogate-Based Inverse Modeling of the Hydrological System: An Adaptive Approach Considering Surrogate Structural Error.

Accelerating Groundwater Data Assimilation with a Gradient‐Free Active Subspace Method

A New Collaborative Neural Network: Using HDMR for Dynamic Structural Adaptation*

Efficient, multimodal, and derivative-free Bayesian inference with Fisher-Rao gradient flows

An adaptive Hessian approximated stochastic gradient MCMC method

High-Dimensional MR Reconstruction Integrating Subspace and Adaptive Generative Models

Exploiting gradients and Hessians in Bayesian optimization and Bayesian quadrature

A novel kriging-improved high-dimensional model representation metamodelling technique for approximating high-dimensional problems

Extending classical surrogate modelling to high-dimensions through supervised dimensionality reduction: a data-driven approach

Dimensionality Reduction for Multi-Fidelity Gaussian Processes Using Bayesian Adaptation

High Dimensional Model Representation as a Glass Box in Supervised Machine Learning

Natural Gradient Hybrid Variational Inference with Application to Deep Mixed Models

Multi-Response Heteroscedastic Gaussian Process Models and Their Inference

An Adaptive Dimension Reduction Estimation Method for High-dimensional Bayesian Optimization

ANOVA-GP Modeling for High-Dimensional Bayesian Inverse Problems

High-Dimensional Regression with Gaussian Mixtures and Partially-Latent Response Variables

Adaptive Bayesian Regression on Data with Low Intrinsic Dimensionality

Sliced gradient-enhanced Kriging for high-dimensional function approximation