Abstract:We study high-probability convergence guarantees of learning on streaming data in the presence of heavy-tailed noise. In the proposed scenario, the model is updated in an online fashion, as new information is observed, without storing any additional data. To combat the heavy-tailed noise, we consider a general framework of nonlinear stochastic gradient descent (SGD), providing several strong results. First, for non-convex costs and component-wise nonlinearities, we establish a convergence rate arbitrarily close to $\mathcal{O}\left(t^{-\frac{1}{4}}\right)$, whose exponent is independent of noise and problem parameters. Second, for strongly convex costs and component-wise nonlinearities, we establish a rate arbitrarily close to $\mathcal{O}\left(t^{-\frac{1}{2}}\right)$ for the weighted average of iterates, with exponent again independent of noise and problem parameters. Finally, for strongly convex costs and a broader class of nonlinearities, we establish convergence of the last iterate, with a rate $\mathcal{O}\left(t^{-\zeta} \right)$, where $\zeta \in (0,1)$ depends on problem parameters, noise and nonlinearity. As we show analytically and numerically, $\zeta$ can be used to inform the preferred choice of nonlinearity for given problem settings. Compared to state-of-the-art, who only consider clipping, require bounded noise moments of order $\eta \in (1,2]$, and establish convergence rates whose exponents go to zero as $\eta \rightarrow 1$, we provide high-probability guarantees for a much broader class of nonlinearities and symmetric density noise, with convergence rates whose exponents are bounded away from zero, even when the noise has finite first moment only. Moreover, in the case of strongly convex functions, we demonstrate analytically and numerically that clipping is not always the optimal nonlinearity, further underlining the value of our general framework.

Deterministic Convergence of Wirtinger-Gradient Methods for Complex-Valued Neural Networks.

Convergence Analysis of General Neural Networks under Almost Periodic Stimuli

High Probability Convergence Bounds for Non-convex Stochastic Gradient Descent with Sub-Weibull Noise

On Convergence of Training Loss Without Reaching Stationary Points

Convergence Analysis of Gradient Algorithms on Riemannian Manifolds Without Curvature Constraints and Application to Riemannian Mass

Convergence analysis of inexact proximal point algorithms on Hadamard manifolds

An Input Weights Dependent Complex-Valued Learning Algorithm Based on Wirtinger Calculus

Stochastic Gradient and Langevin Processes

Convergence of Online Gradient Algorithm with Stochastic Inputs for Pi-Sigma Neural Networks

Convergence of Hyperbolic Neural Networks under Riemannian Stochastic Gradient Descent

A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks

A Convergence Theory Towards Practical Over-parameterized Deep Neural Networks

Convergence of SGD for Training Neural Networks with Sliced Wasserstein Losses

Convergence Analysis of Two-layer Neural Networks with ReLU Activation

On the convergence of gradient descent for two layer neural networks

Convergence Analysis of a Class of Nonsmooth Gradient Systems.

Non-asymptotic convergence analysis of the stochastic gradient Hamiltonian Monte Carlo algorithm with discontinuous stochastic gradient with applications to training of ReLU neural networks

High-probability Convergence Bounds for Nonlinear Stochastic Gradient Descent Under Heavy-tailed Noise

Convergence dynamics of stochastic Cohen-Grossberg neural networks with unbounded distributed delays

Quantitative convergence of trained quantum neural networks to a Gaussian process

On the Convergence of Gradient Descent for Large Learning Rates