Abstract:The rise of artificial intelligence (AI) hinges on the efficient training of modern deep neural networks (DNNs) for non-convex optimization and uncertainty quantification, which boils down to a non-convex Bayesian learning problem. A standard tool to handle the problem is Langevin Monte Carlo, which proposes to approximate the posterior distribution with theoretical guarantees. In this thesis, we start with the replica exchange Langevin Monte Carlo (also known as parallel tempering), which proposes appropriate swaps between exploration and exploitation to achieve accelerations. However, the naïve extension of swaps to big data problems leads to a large bias, and bias-corrected swaps are required. Such a mechanism leads to few effective swaps and insignificant accelerations. To alleviate this issue, we first propose a control variates method to reduce the variance of noisy energy estimators and show a potential to accelerate the exponential convergence. We also present the population-chain replica exchange based on non-reversibility and obtain an optimal round-trip rate for deep learning. In the second part of the thesis, we study scalable dynamic importance sampling algorithms based on stochastic approximation. Traditional dynamic importance sampling algorithms have achieved success, however, the lack of scalability has greatly limited their extensions to big data. To handle this scalability issue, we resolve the vanishing gradient problem and propose two dynamic importance sampling algorithms. Theoretically, we establish the stability condition for the underlying ordinary differential equation (ODE) system and guarantee the asymptotic convergence of the latent variable to the desired fixed point. Interestingly, such a result still holds given non-convex energy landscapes.

Optimization by Stochastic Continuation

Stochastic Successive Convex Approximation for Non-Convex Constrained Stochastic Optimization

Online Successive Convex Approximation for Two-Stage Stochastic Nonconvex Optimization

Efficient stochastic optimisation by unadjusted Langevin Monte Carlo. Application to maximum marginal likelihood and empirical Bayesian estimation

Stochastic Approximation Beyond Gradient for Signal Processing and Machine Learning

Optimal Stochastic Approximation Algorithms for Strongly Convex Stochastic Composite Optimization, II: Shrinking Procedures and Optimal Algorithms

Accelerated stochastic approximation with state-dependent noise

Optimal Stochastic Approximation Algorithms for Strongly Convex Stochastic Composite Optimization I: A Generic Algorithmic Framework

An optimal method for stochastic composite optimization

Robust Stochastic Approximation Approach to Stochastic Programming

Constrained Stochastic Recursive Momentum Successive Convex Approximation

Non-convex Bayesian Learning via Stochastic Gradient Markov Chain Monte Carlo

Stochastic Approximation Proximal Subgradient Method for Stochastic Convex-Concave Minimax Optimization

Optimization with Discrete Simultaneous Perturbation Stochastic Approximation Using Noisy Loss Function Measurements

Annealing Optimization for Progressive Learning With Stochastic Approximation

An Algebraically Converging Stochastic Gradient Descent Algorithm for Global Optimization

Optimization by linear kinetic equations and mean-field Langevin dynamics

Markovian Foundations for Quasi-Stochastic Approximation in Two Timescales: Extended Version

Re-use of samples in stochastic annealing

Stochastic Simulated Quantum Annealing for Fast Solution of Combinatorial Optimization Problems