Abstract:We study high-probability convergence guarantees of learning on streaming data in the presence of heavy-tailed noise. In the proposed scenario, the model is updated in an online fashion, as new information is observed, without storing any additional data. To combat the heavy-tailed noise, we consider a general framework of nonlinear stochastic gradient descent (SGD), providing several strong results. First, for non-convex costs and component-wise nonlinearities, we establish a convergence rate arbitrarily close to $\mathcal{O}\left(t^{-\frac{1}{4}}\right)$, whose exponent is independent of noise and problem parameters. Second, for strongly convex costs and component-wise nonlinearities, we establish a rate arbitrarily close to $\mathcal{O}\left(t^{-\frac{1}{2}}\right)$ for the weighted average of iterates, with exponent again independent of noise and problem parameters. Finally, for strongly convex costs and a broader class of nonlinearities, we establish convergence of the last iterate, with a rate $\mathcal{O}\left(t^{-\zeta} \right)$, where $\zeta \in (0,1)$ depends on problem parameters, noise and nonlinearity. As we show analytically and numerically, $\zeta$ can be used to inform the preferred choice of nonlinearity for given problem settings. Compared to state-of-the-art, who only consider clipping, require bounded noise moments of order $\eta \in (1,2]$, and establish convergence rates whose exponents go to zero as $\eta \rightarrow 1$, we provide high-probability guarantees for a much broader class of nonlinearities and symmetric density noise, with convergence rates whose exponents are bounded away from zero, even when the noise has finite first moment only. Moreover, in the case of strongly convex functions, we demonstrate analytically and numerically that clipping is not always the optimal nonlinearity, further underlining the value of our general framework.

Local KL Convergence Rate for Stein Variational Gradient Descent with Reweighted Kernel

Improved Finite-Particle Convergence Rates for Stein Variational Gradient Descent

Gradient-free Stein Variational Gradient Descent with Kernel Approximation

The Stein-log-Sobolev inequality and the exponential rate of convergence for the continuous Stein variational gradient descent method

Regularized Stein Variational Gradient Flow

Stein Variational Gradient Descent with Multiple Kernel

Stein Variational Gradient Descent with Local Approximations

Convergence Rates for Stochastic Approximation: Biased Noise with Unbounded Variance, and Applications

Local smoothness in variance reduced optimization

Stochastic gradient descent, weighted sampling, and the randomized Kaczmarz algorithm

Rate of convergence of the smoothed empirical Wasserstein distance

On the mean-field limit for Stein variational gradient descent: stability and multilevel approximation

High Probability Convergence Bounds for Non-convex Stochastic Gradient Descent with Sub-Weibull Noise

High-probability Convergence Bounds for Nonlinear Stochastic Gradient Descent Under Heavy-tailed Noise

Convergence of a Normal Map-based Prox-SGD Method under the KL Inequality

Global Convergence of Langevin Dynamics Based Algorithms for Nonconvex Optimization

Augmented Message Passing Stein Variational Gradient Descent

Linear Convergence Rate in Convex Setup is Possible! Gradient Descent Method Variants under $(L_0,L_1)$-Smoothness

Convergence of Unregularized Online Learning Algorithms

Uniform Convergence Rate of the Kernel Density Estimator Adaptive to Intrinsic Volume Dimension

Adaptive Variance Reducing for Stochastic Gradient Descent.