Abstract:We study high-probability convergence guarantees of learning on streaming data in the presence of heavy-tailed noise. In the proposed scenario, the model is updated in an online fashion, as new information is observed, without storing any additional data. To combat the heavy-tailed noise, we consider a general framework of nonlinear stochastic gradient descent (SGD), providing several strong results. First, for non-convex costs and component-wise nonlinearities, we establish a convergence rate arbitrarily close to $\mathcal{O}\left(t^{-\frac{1}{4}}\right)$, whose exponent is independent of noise and problem parameters. Second, for strongly convex costs and component-wise nonlinearities, we establish a rate arbitrarily close to $\mathcal{O}\left(t^{-\frac{1}{2}}\right)$ for the weighted average of iterates, with exponent again independent of noise and problem parameters. Finally, for strongly convex costs and a broader class of nonlinearities, we establish convergence of the last iterate, with a rate $\mathcal{O}\left(t^{-\zeta} \right)$, where $\zeta \in (0,1)$ depends on problem parameters, noise and nonlinearity. As we show analytically and numerically, $\zeta$ can be used to inform the preferred choice of nonlinearity for given problem settings. Compared to state-of-the-art, who only consider clipping, require bounded noise moments of order $\eta \in (1,2]$, and establish convergence rates whose exponents go to zero as $\eta \rightarrow 1$, we provide high-probability guarantees for a much broader class of nonlinearities and symmetric density noise, with convergence rates whose exponents are bounded away from zero, even when the noise has finite first moment only. Moreover, in the case of strongly convex functions, we demonstrate analytically and numerically that clipping is not always the optimal nonlinearity, further underlining the value of our general framework.

Embedding generalization within the learning dynamics: An approach based-on sample path large deviation theory

Generalization Bounds for Stochastic Gradient Langevin Dynamics: A Unified View Via Information Leakage Analysis

On improving generalization in a class of learning problems with the method of small parameters for weakly-controlled optimal gradient systems

On Generalization Error Bounds of Noisy Gradient Methods for Non-Convex Learning

Generalization Bounds for Gradient Methods via Discrete and Continuous Prior

A new perspective on the learning dynamics for a class of learning problems via averaged gradient systems coupled with diffusion-transmutation processes

Generalization bounds of sgld for non-convex learning: Two theoretical viewpoints

Local Optimality and Generalization Guarantees for the Langevin Algorithm via Empirical Metastability

Generalization Bounds of SGLD for Non-convex Learning: Two Theoretical Viewpoints.

Towards Optimal Problem Dependent Generalization Error Bounds in Statistical Learning Theory

Towards Understanding Generalization Via Decomposing Excess Risk Dynamics

Finite Sample Analysis and Bounds of Generalization Error of Gradient Descent in In-Context Linear Regression

A Generalization Result for Convergence in Learning-to-Optimize

Estimating Generalization Performance Along the Trajectory of Proximal SGD in Robust Regression

Langevin Dynamics: A Unified Perspective on Optimization via Lyapunov Potentials

High Probability Guarantees for Nonconvex Stochastic Gradient Descent with Heavy Tails.

Optimal sampling for stochastic and natural gradient descent

High-probability Convergence Bounds for Nonlinear Stochastic Gradient Descent Under Heavy-tailed Noise

A generalization of regularized dual averaging and its dynamics

A General Framework for Analyzing Stochastic Dynamics in Learning Algorithms

Towards Understanding Generalization via Decomposing Excess Risk Dynamics