Abstract:The rise of artificial intelligence (AI) hinges on the efficient training of modern deep neural networks (DNNs) for non-convex optimization and uncertainty quantification, which boils down to a non-convex Bayesian learning problem. A standard tool to handle the problem is Langevin Monte Carlo, which proposes to approximate the posterior distribution with theoretical guarantees. In this thesis, we start with the replica exchange Langevin Monte Carlo (also known as parallel tempering), which proposes appropriate swaps between exploration and exploitation to achieve accelerations. However, the naïve extension of swaps to big data problems leads to a large bias, and bias-corrected swaps are required. Such a mechanism leads to few effective swaps and insignificant accelerations. To alleviate this issue, we first propose a control variates method to reduce the variance of noisy energy estimators and show a potential to accelerate the exponential convergence. We also present the population-chain replica exchange based on non-reversibility and obtain an optimal round-trip rate for deep learning. In the second part of the thesis, we study scalable dynamic importance sampling algorithms based on stochastic approximation. Traditional dynamic importance sampling algorithms have achieved success, however, the lack of scalability has greatly limited their extensions to big data. To handle this scalability issue, we resolve the vanishing gradient problem and propose two dynamic importance sampling algorithms. Theoretically, we establish the stability condition for the underlying ordinary differential equation (ODE) system and guarantee the asymptotic convergence of the latent variable to the desired fixed point. Interestingly, such a result still holds given non-convex energy landscapes.

Optimization under Uncertainty: A Perspective of Soft Computing

Probabilistic Computation with Emerging Covariance: Towards Efficient Uncertainty Quantification

On Uncertainty Quantification for Near-Bayes Optimal Algorithms

Uncertainty Quantification for Sparse Deep Learning

A Direct Coupled Solution Methodology for Efficient Robust Optimizations of Inverse Problems under Uncertainty

A Study of Bayesian Neural Network Surrogates for Bayesian Optimization

Learning Solutions of Stochastic Optimization Problems with Bayesian Neural Networks

Uncertainty Quantification for Bayesian Optimization

Density-Softmax: Efficient Test-time Model for Uncertainty Estimation and Robustness under Distribution Shifts

Deterministic Variational Inference for Robust Bayesian Neural Networks

Function Space Particle Optimization for Bayesian Neural Networks

Variational Inference for Bayesian Neural Networks under Model and Parameter Uncertainty

Non-convex Bayesian Learning via Stochastic Gradient Markov Chain Monte Carlo

Alpha-divergence minimization with mixed variational posterior for Bayesian neural networks and its robustness against adversarial examples

Implicit Generative Prior for Bayesian Neural Networks

Bayesian Deep Convolutional Encoder-Decoder Networks for Surrogate Modeling and Uncertainty Quantification

Combining Model and Parameter Uncertainty in Bayesian Neural Networks

A Bayesian Risk Approach to Data-driven Stochastic Optimization: Formulations and Asymptotics

IB-UQ: Information bottleneck based uncertainty quantification for neural function regression and neural operator learning

Pseudo-Bayesian Optimization

Process Optimization with Consideration of Uncertainties-an Overview