Abstract:We study high-probability convergence guarantees of learning on streaming data in the presence of heavy-tailed noise. In the proposed scenario, the model is updated in an online fashion, as new information is observed, without storing any additional data. To combat the heavy-tailed noise, we consider a general framework of nonlinear stochastic gradient descent (SGD), providing several strong results. First, for non-convex costs and component-wise nonlinearities, we establish a convergence rate arbitrarily close to $\mathcal{O}\left(t^{-\frac{1}{4}}\right)$, whose exponent is independent of noise and problem parameters. Second, for strongly convex costs and component-wise nonlinearities, we establish a rate arbitrarily close to $\mathcal{O}\left(t^{-\frac{1}{2}}\right)$ for the weighted average of iterates, with exponent again independent of noise and problem parameters. Finally, for strongly convex costs and a broader class of nonlinearities, we establish convergence of the last iterate, with a rate $\mathcal{O}\left(t^{-\zeta} \right)$, where $\zeta \in (0,1)$ depends on problem parameters, noise and nonlinearity. As we show analytically and numerically, $\zeta$ can be used to inform the preferred choice of nonlinearity for given problem settings. Compared to state-of-the-art, who only consider clipping, require bounded noise moments of order $\eta \in (1,2]$, and establish convergence rates whose exponents go to zero as $\eta \rightarrow 1$, we provide high-probability guarantees for a much broader class of nonlinearities and symmetric density noise, with convergence rates whose exponents are bounded away from zero, even when the noise has finite first moment only. Moreover, in the case of strongly convex functions, we demonstrate analytically and numerically that clipping is not always the optimal nonlinearity, further underlining the value of our general framework.

A Tight Convergence Analysis for Stochastic Gradient Descent with Delayed Updates

Convergence Analysis of Asynchronous Stochastic Recursive Gradient Algorithms

High-probability Convergence Bounds for Nonlinear Stochastic Gradient Descent Under Heavy-tailed Noise

Faster Stochastic Optimization with Arbitrary Delays via Asynchronous Mini-Batching

Delayed Stochastic Algorithms for Distributed Weakly Convex Optimization

Delayed Algorithms for Distributed Stochastic Weakly Convex Optimization

Tight Nonparametric Convergence Rates for Stochastic Gradient Descent under the Noiseless Linear Model

A Sharp Convergence Rate for the Asynchronous Stochastic Gradient Descent

Convergence Analysis of Adaptive Gradient Methods under Refined Smoothness and Noise Assumptions

Distributed Stochastic Optimization with Random Communication and Computational Delays: Optimal Policies and Performance Analysis

Convergence in High Probability of Distributed Stochastic Gradient Descent Algorithms

Convergence in quadratic mean of averaged stochastic gradient algorithms without strong convexity nor bounded gradient

High Probability Convergence Bounds for Non-convex Stochastic Gradient Descent with Sub-Weibull Noise

Convergence Rates for Stochastic Approximation: Biased Noise with Unbounded Variance, and Applications

A Continuous-Time Analysis of Distributed Stochastic Gradient

Local Quadratic Convergence of Stochastic Gradient Descent with Adaptive Step Size

Convergence and concentration properties of constant step-size SGD through Markov chains

On Convergence Rate of Distributed Stochastic Gradient Algorithm for Convex Optimization with Inequality Constraints.

Convergence Analysis of Accelerated Stochastic Gradient Descent under the Growth Condition

Stochastic Approximation with Delayed Updates: Finite-Time Rates under Markovian Sampling

Convergence of Constant Step Stochastic Gradient Descent for Non-Smooth Non-Convex Functions