Abstract:We study high-probability convergence guarantees of learning on streaming data in the presence of heavy-tailed noise. In the proposed scenario, the model is updated in an online fashion, as new information is observed, without storing any additional data. To combat the heavy-tailed noise, we consider a general framework of nonlinear stochastic gradient descent (SGD), providing several strong results. First, for non-convex costs and component-wise nonlinearities, we establish a convergence rate arbitrarily close to $\mathcal{O}\left(t^{-\frac{1}{4}}\right)$, whose exponent is independent of noise and problem parameters. Second, for strongly convex costs and component-wise nonlinearities, we establish a rate arbitrarily close to $\mathcal{O}\left(t^{-\frac{1}{2}}\right)$ for the weighted average of iterates, with exponent again independent of noise and problem parameters. Finally, for strongly convex costs and a broader class of nonlinearities, we establish convergence of the last iterate, with a rate $\mathcal{O}\left(t^{-\zeta} \right)$, where $\zeta \in (0,1)$ depends on problem parameters, noise and nonlinearity. As we show analytically and numerically, $\zeta$ can be used to inform the preferred choice of nonlinearity for given problem settings. Compared to state-of-the-art, who only consider clipping, require bounded noise moments of order $\eta \in (1,2]$, and establish convergence rates whose exponents go to zero as $\eta \rightarrow 1$, we provide high-probability guarantees for a much broader class of nonlinearities and symmetric density noise, with convergence rates whose exponents are bounded away from zero, even when the noise has finite first moment only. Moreover, in the case of strongly convex functions, we demonstrate analytically and numerically that clipping is not always the optimal nonlinearity, further underlining the value of our general framework.

On the Generalization Ability of Online Gradient Descent Algorithm under the Quadratic Growth Condition.

On Generalization Error Bounds of Noisy Gradient Methods for Non-Convex Learning

Fast and Strong Convergence of Online Learning Algorithms.

Convergence of Unregularized Online Learning Algorithms

Generalization Bounds for Gradient Methods via Discrete and Continuous Prior

Gradient Methods with Online Scaling

Generalization Bounds of SGLD for Non-convex Learning: Two Theoretical Viewpoints.

High-probability Convergence Bounds for Nonlinear Stochastic Gradient Descent Under Heavy-tailed Noise

A generalization of regularized dual averaging and its dynamics

Convergence Theory of Generalized Distributed Subgradient Method with Random Quantization

Nonlinear Stochastic Gradient Descent and Heavy-tailed Noise: A Unified Framework and High-probability Guarantees

The Optimality of (Accelerated) SGD for High-Dimensional Quadratic Optimization

Lower Generalization Bounds for GD and SGD in Smooth Stochastic Convex Optimization

Demystifying SGD with Doubly Stochastic Gradients

Boosting the Confidence of Generalization for L2-Stable Randomized Learning Algorithms

Stability and Generalization of the Decentralized Stochastic Gradient Descent Ascent Algorithm

Towards Understanding the Generalizability of Delayed Stochastic Gradient Descent

The Dimension Strikes Back with Gradients: Generalization of Gradient Methods in Stochastic Convex Optimization

Gradient-Variation Online Learning under Generalized Smoothness

On the Privacy of Noisy Stochastic Gradient Descent for Convex Optimization

Convergence Analysis of Accelerated Stochastic Gradient Descent under the Growth Condition