Abstract:Recently, several studies consider the stochastic optimization problem but in a heavy-tailed noise regime, i.e., the difference between the stochastic gradient and the true gradient is assumed to have a finite $p$-th moment (say being upper bounded by $\sigma^{p}$ for some $\sigma\geq0$) where $p\in(1,2]$, which not only generalizes the traditional finite variance assumption ($p=2$) but also has been observed in practice for several different tasks. Under this challenging assumption, lots of new progress has been made for either convex or nonconvex problems, however, most of which only consider smooth objectives. In contrast, people have not fully explored and well understood this problem when functions are nonsmooth. This paper aims to fill this crucial gap by providing a comprehensive analysis of stochastic nonsmooth convex optimization with heavy-tailed noises. We revisit a simple clipping-based algorithm, whereas, which is only proved to converge in expectation but under the additional strong convexity assumption. Under appropriate choices of parameters, for both convex and strongly convex functions, we not only establish the first high-probability rates but also give refined in-expectation bounds compared with existing works. Remarkably, all of our results are optimal (or nearly optimal up to logarithmic factors) with respect to the time horizon $T$ even when $T$ is unknown in advance. Additionally, we show how to make the algorithm parameter-free with respect to $\sigma$, in other words, the algorithm can still guarantee convergence without any prior knowledge of $\sigma$. Furthermore, an initial distance adaptive convergence rate is provided if $\sigma$ is assumed to be known.

On deviation probabilities in non-parametric regression with heavy-tailed noise

High Probability Bounds for Stochastic Subgradient Schemes with Heavy Tailed Noise

Mean estimation and regression under heavy-tailed distributions--a survey

On Regression in Extreme Regions

\Ell_1-Regression with Heavy-tailed Distributions.

High-dimensional robust regression under heavy-tailed data: Asymptotics and Universality

Mean-Square Analysis of Discretized Itô Diffusions for Heavy-tailed Sampling

Stochastic Nonsmooth Convex Optimization with Heavy-Tailed Noises: High-Probability Bound, In-Expectation Rate and Initial Distance Adaptation

Nonparametric smoothing for extremal quantile regression with heavy tailed distributions

Large Deviations for High Minima of Gaussian Processes with Nonnegatively Correlated Increments

Lower bounds for the trade-off between bias and mean absolute deviation

Quantitative bounds for large deviations of heavy tailed random variables

High Probability Convergence Bounds for Non-convex Stochastic Gradient Descent with Sub-Weibull Noise

Large Deviations for Small Noise Diffusions Over Long Time

Tight Nonparametric Convergence Rates for Stochastic Gradient Descent under the Noiseless Linear Model

L G ] 2 5 O ct 2 01 8 l 1-regression with Heavy-tailed Distributions

High-Probability Complexity Bounds for Non-smooth Stochastic Convex Optimization with Heavy-Tailed Noise

Query Complexity of Least Absolute Deviation Regression via Robust Uniform Convergence

Large deviations for a class of multivariate heavy-tailed risk processes used in insurance and finance

Sparse Linear Regression when Noises and Covariates are Heavy-Tailed and Contaminated by Outliers

Mean Estimation in Banach Spaces Under Infinite Variance and Martingale Dependence