Abstract:The rise of artificial intelligence (AI) hinges on the efficient training of modern deep neural networks (DNNs) for non-convex optimization and uncertainty quantification, which boils down to a non-convex Bayesian learning problem. A standard tool to handle the problem is Langevin Monte Carlo, which proposes to approximate the posterior distribution with theoretical guarantees. In this thesis, we start with the replica exchange Langevin Monte Carlo (also known as parallel tempering), which proposes appropriate swaps between exploration and exploitation to achieve accelerations. However, the naïve extension of swaps to big data problems leads to a large bias, and bias-corrected swaps are required. Such a mechanism leads to few effective swaps and insignificant accelerations. To alleviate this issue, we first propose a control variates method to reduce the variance of noisy energy estimators and show a potential to accelerate the exponential convergence. We also present the population-chain replica exchange based on non-reversibility and obtain an optimal round-trip rate for deep learning. In the second part of the thesis, we study scalable dynamic importance sampling algorithms based on stochastic approximation. Traditional dynamic importance sampling algorithms have achieved success, however, the lack of scalability has greatly limited their extensions to big data. To handle this scalability issue, we resolve the vanishing gradient problem and propose two dynamic importance sampling algorithms. Theoretically, we establish the stability condition for the underlying ordinary differential equation (ODE) system and guarantee the asymptotic convergence of the latent variable to the desired fixed point. Interestingly, such a result still holds given non-convex energy landscapes.

Nonsmooth Nonconvex Stochastic Heavy Ball

Accelerated Convergence of Stochastic Heavy Ball Method under Anisotropic Gradient Noise

Unified Convergence Analysis of Stochastic Momentum Methods for Convex and Non-convex Optimization

On the fast convergence of minibatch heavy ball momentum

Heavy-ball Algorithms Always Escape Saddle Points

Stochastic Bregman Subgradient Methods for Nonsmooth Nonconvex Optimization Problems

(Accelerated) Noise-adaptive Stochastic Heavy-Ball Momentum

A New Variant of Stochastic Heavy Ball Optimization Method for Deep Learning

Convergence of Constant Step Stochastic Gradient Descent for Non-Smooth Non-Convex Functions

Subgradient sampling for nonsmooth nonconvex minimization

Convergence analysis of a stochastic heavy-ball method for linear ill-posed problems

Nonconvex Stochastic Bregman Proximal Gradient Method with Application to Deep Learning

A Stochastic Semismooth Newton Method for Nonsmooth Nonconvex Optimization.

Heavy Ball Momentum for Non-Strongly Convex Optimization

Decentralized Stochastic Subgradient Methods for Nonsmooth Nonconvex Optimization

Non-convex Bayesian Learning via Stochastic Gradient Markov Chain Monte Carlo

High-Probability Bound for Non-Smooth Non-Convex Stochastic Optimization with Heavy Tails

High Probability Convergence Bounds for Non-convex Stochastic Gradient Descent with Sub-Weibull Noise

Primitive Heavy-ball Dynamics Achieves $O(\varepsilon^{-7/4})$ Convergence for Nonconvex Optimization

Stochastic Subgradient Methods with Guaranteed Global Stability in Nonsmooth Nonconvex Optimization

A note on the convergence of deterministic gradient sampling in nonsmooth optimization