Abstract:We study high-probability convergence guarantees of learning on streaming data in the presence of heavy-tailed noise. In the proposed scenario, the model is updated in an online fashion, as new information is observed, without storing any additional data. To combat the heavy-tailed noise, we consider a general framework of nonlinear stochastic gradient descent (SGD), providing several strong results. First, for non-convex costs and component-wise nonlinearities, we establish a convergence rate arbitrarily close to $\mathcal{O}\left(t^{-\frac{1}{4}}\right)$, whose exponent is independent of noise and problem parameters. Second, for strongly convex costs and component-wise nonlinearities, we establish a rate arbitrarily close to $\mathcal{O}\left(t^{-\frac{1}{2}}\right)$ for the weighted average of iterates, with exponent again independent of noise and problem parameters. Finally, for strongly convex costs and a broader class of nonlinearities, we establish convergence of the last iterate, with a rate $\mathcal{O}\left(t^{-\zeta} \right)$, where $\zeta \in (0,1)$ depends on problem parameters, noise and nonlinearity. As we show analytically and numerically, $\zeta$ can be used to inform the preferred choice of nonlinearity for given problem settings. Compared to state-of-the-art, who only consider clipping, require bounded noise moments of order $\eta \in (1,2]$, and establish convergence rates whose exponents go to zero as $\eta \rightarrow 1$, we provide high-probability guarantees for a much broader class of nonlinearities and symmetric density noise, with convergence rates whose exponents are bounded away from zero, even when the noise has finite first moment only. Moreover, in the case of strongly convex functions, we demonstrate analytically and numerically that clipping is not always the optimal nonlinearity, further underlining the value of our general framework.

Local SGD for Near-Quadratic Problems: Improving Convergence under Unconstrained Noise Conditions

Decentralized SGD with Asynchronous, Local and Quantized Updates

Accelerated stochastic approximation with state-dependent noise

Error compensated quantized SGD and its applications to large-scale distributed optimization

Beyond Convexity: Stochastic Quasi-Convex Optimization

A Unified Theory of Decentralized SGD with Changing Topology and Local Updates

Asynchronous Decentralized SGD with Quantized and Local Updates.

Local Quadratic Convergence of Stochastic Gradient Descent with Adaptive Step Size

The Optimality of (Accelerated) SGD for High-Dimensional Quadratic Optimization

Convergence Rates for Stochastic Approximation: Biased Noise with Unbounded Variance, and Applications

The Limits and Potentials of Local SGD for Distributed Heterogeneous Learning with Intermittent Communication

Distributed Subgradient Method with Random Quantization and Flexible Weights: Convergence Analysis.

Convergence Theory of Generalized Distributed Subgradient Method with Random Quantization

A Linear Speedup Analysis of Distributed Deep Learning with Sparse and Quantized Communication.

Eigen-componentwise convergence of SGD on quadratic programming

Gradient-tracking based Distributed Optimization with Guaranteed Optimality under Noisy Information Sharing

Accelerated zero-order SGD under high-order smoothness and overparameterized regime

Scaling up stochastic gradient descent for non-convex optimisation

Variance-Reduced Stochastic Quasi-Newton Methods for Decentralized Learning: Part II

High-probability Convergence Bounds for Nonlinear Stochastic Gradient Descent Under Heavy-tailed Noise