Abstract:We study high-probability convergence guarantees of learning on streaming data in the presence of heavy-tailed noise. In the proposed scenario, the model is updated in an online fashion, as new information is observed, without storing any additional data. To combat the heavy-tailed noise, we consider a general framework of nonlinear stochastic gradient descent (SGD), providing several strong results. First, for non-convex costs and component-wise nonlinearities, we establish a convergence rate arbitrarily close to $\mathcal{O}\left(t^{-\frac{1}{4}}\right)$, whose exponent is independent of noise and problem parameters. Second, for strongly convex costs and component-wise nonlinearities, we establish a rate arbitrarily close to $\mathcal{O}\left(t^{-\frac{1}{2}}\right)$ for the weighted average of iterates, with exponent again independent of noise and problem parameters. Finally, for strongly convex costs and a broader class of nonlinearities, we establish convergence of the last iterate, with a rate $\mathcal{O}\left(t^{-\zeta} \right)$, where $\zeta \in (0,1)$ depends on problem parameters, noise and nonlinearity. As we show analytically and numerically, $\zeta$ can be used to inform the preferred choice of nonlinearity for given problem settings. Compared to state-of-the-art, who only consider clipping, require bounded noise moments of order $\eta \in (1,2]$, and establish convergence rates whose exponents go to zero as $\eta \rightarrow 1$, we provide high-probability guarantees for a much broader class of nonlinearities and symmetric density noise, with convergence rates whose exponents are bounded away from zero, even when the noise has finite first moment only. Moreover, in the case of strongly convex functions, we demonstrate analytically and numerically that clipping is not always the optimal nonlinearity, further underlining the value of our general framework.

Gradient Normalization Provably Benefits Nonconvex SGD under Heavy-Tailed Noise

From Gradient Clipping to Normalization for Heavy Tailed SGD

Gradient Clipping Improves AdaGrad when the Noise Is Heavy-Tailed

High Probability Guarantees for Nonconvex Stochastic Gradient Descent with Heavy Tails.

Nonlinear Stochastic Gradient Descent and Heavy-tailed Noise: A Unified Framework and High-probability Guarantees

High-probability Convergence Bounds for Nonlinear Stochastic Gradient Descent Under Heavy-tailed Noise

Improved Convergence in High Probability of Clipped Gradient Methods with Heavy Tails

Algorithms with Gradient Clipping for Stochastic Optimization with Heavy-Tailed Noise

Smoothed Gradient Clipping and Error Feedback for Decentralized Optimization under Symmetric Heavy-Tailed Noise

Noisy Truncated SGD: Optimization and Generalization

High Probability Analysis for Non-Convex Stochastic Optimization with Clipping

Revisiting Gradient Clipping: Stochastic bias and tight convergence guarantees

On the Convergence and Improvement of Stochastic Normalized Gradient Descent

Stability and Convergence of Stochastic Gradient Clipping: Beyond Lipschitz Continuity and Smoothness

To Clip or not to Clip: the Dynamics of SGD with Gradient Clipping in High-Dimensions

Improved Analysis of Clipping Algorithms for Non-convex Optimization

Emergence of heavy tails in homogenized stochastic gradient descent

Global Convergence of Noisy Gradient Descent.

High Probability Convergence Bounds for Non-convex Stochastic Gradient Descent with Sub-Weibull Noise

Clip Body and Tail Separately: High Probability Guarantees for DPSGD with Heavy Tails

Accelerated Convergence of Stochastic Heavy Ball Method under Anisotropic Gradient Noise