Abstract:We study high-probability convergence guarantees of learning on streaming data in the presence of heavy-tailed noise. In the proposed scenario, the model is updated in an online fashion, as new information is observed, without storing any additional data. To combat the heavy-tailed noise, we consider a general framework of nonlinear stochastic gradient descent (SGD), providing several strong results. First, for non-convex costs and component-wise nonlinearities, we establish a convergence rate arbitrarily close to $\mathcal{O}\left(t^{-\frac{1}{4}}\right)$, whose exponent is independent of noise and problem parameters. Second, for strongly convex costs and component-wise nonlinearities, we establish a rate arbitrarily close to $\mathcal{O}\left(t^{-\frac{1}{2}}\right)$ for the weighted average of iterates, with exponent again independent of noise and problem parameters. Finally, for strongly convex costs and a broader class of nonlinearities, we establish convergence of the last iterate, with a rate $\mathcal{O}\left(t^{-\zeta} \right)$, where $\zeta \in (0,1)$ depends on problem parameters, noise and nonlinearity. As we show analytically and numerically, $\zeta$ can be used to inform the preferred choice of nonlinearity for given problem settings. Compared to state-of-the-art, who only consider clipping, require bounded noise moments of order $\eta \in (1,2]$, and establish convergence rates whose exponents go to zero as $\eta \rightarrow 1$, we provide high-probability guarantees for a much broader class of nonlinearities and symmetric density noise, with convergence rates whose exponents are bounded away from zero, even when the noise has finite first moment only. Moreover, in the case of strongly convex functions, we demonstrate analytically and numerically that clipping is not always the optimal nonlinearity, further underlining the value of our general framework.

Convergence of Langevin-simulated annealing algorithms with multiplicative noise

Gradual convergence for Langevin dynamics on a degenerate potential

Global Convergence of Langevin Dynamics Based Algorithms for Nonconvex Optimization

Explicit convergence rates of underdamped Langevin dynamics under weighted and weak Poincaré--Lions inequalities

Unadjusted Langevin Algorithms for SDEs with Hoelder Drift

Convergence of Unadjusted Langevin in High Dimensions: Delocalization of Bias

Langevin Dynamics: A Unified Perspective on Optimization via Lyapunov Potentials

When is the Convergence Time of Langevin Algorithms Dimension Independent? A Composite Optimization Viewpoint

Fast and Accurate Numerical Integration of the Langevin Equation with Multiplicative Gaussian White Noise

Convergence of Relative Entropy for Euler–Maruyama Scheme to Stochastic Differential Equations with Additive Noise

Optimization by linear kinetic equations and mean-field Langevin dynamics

Provable Benefit of Annealed Langevin Monte Carlo for Non-log-concave Sampling

High Probability Convergence Bounds for Non-convex Stochastic Gradient Descent with Sub-Weibull Noise

Concentration of Contractive Stochastic Approximation: Additive and Multiplicative Noise

Temporal approximation of stochastic evolution equations with irregular nonlinearities

Projected Langevin Monte Carlo algorithms in non-convex and super-linear setting

Convergence Of The Unadjusted Langevin Algorithm For Discontinuous Gradients

High-probability Convergence Bounds for Nonlinear Stochastic Gradient Descent Under Heavy-tailed Noise

Accelerated convergence to equilibrium and reduced asymptotic variance for Langevin dynamics using Stratonovich perturbations

Local Optimality and Generalization Guarantees for the Langevin Algorithm via Empirical Metastability