Abstract:Recently, several studies consider the stochastic optimization problem but in a heavy-tailed noise regime, i.e., the difference between the stochastic gradient and the true gradient is assumed to have a finite $p$-th moment (say being upper bounded by $\sigma^{p}$ for some $\sigma\geq0$) where $p\in(1,2]$, which not only generalizes the traditional finite variance assumption ($p=2$) but also has been observed in practice for several different tasks. Under this challenging assumption, lots of new progress has been made for either convex or nonconvex problems, however, most of which only consider smooth objectives. In contrast, people have not fully explored and well understood this problem when functions are nonsmooth. This paper aims to fill this crucial gap by providing a comprehensive analysis of stochastic nonsmooth convex optimization with heavy-tailed noises. We revisit a simple clipping-based algorithm, whereas, which is only proved to converge in expectation but under the additional strong convexity assumption. Under appropriate choices of parameters, for both convex and strongly convex functions, we not only establish the first high-probability rates but also give refined in-expectation bounds compared with existing works. Remarkably, all of our results are optimal (or nearly optimal up to logarithmic factors) with respect to the time horizon $T$ even when $T$ is unknown in advance. Additionally, we show how to make the algorithm parameter-free with respect to $\sigma$, in other words, the algorithm can still guarantee convergence without any prior knowledge of $\sigma$. Furthermore, an initial distance adaptive convergence rate is provided if $\sigma$ is assumed to be known.

Extract the Information from Big Data with Randomly Distributed Noise

Low Rank Matrix Recovery with Adversarial Sparse Noise

A Tikhonov Regularization Based Algorithm for Scattered Data with Random Noise

A Big Data Processing Technique Based on Tikhonov Regularization

Sparse Estimation Via ℓ_q Optimization Method in High-Dimensional Linear Regression

On the Asymptotical Regularization for Linear Inverse Problems in Presence of White Noise

Robust Estimation for Image Noise Based on Eigenvalue Distributions of Large Sample Covariance Matrices.

On Learning the Optimal Regularization Parameter in Inverse Problems

An Approach of Regularization Parameter Estimation for Sparse Signal Recovery

Regularized deep learning for unsupervised random noise attenuation in poststack seismic data

Matrix recovery from nonconvex regularized least absolute deviations

Density-driven Regularization for Out-of-distribution Detection

Regularization by Nonlinear Noise for PDEs: Well-posedness and Finite Time Extinction

Stochastic Nonsmooth Convex Optimization with Heavy-Tailed Noises: High-Probability Bound, In-Expectation Rate and Initial Distance Adaptation

Stochastic Convergence of Regularized Solutions and Their Finite Element Approximations to Inverse Source Problems

Noisy recovery from random linear observations: Sharp minimax rates under elliptical constraints

Noisy Data Collection Towards Diversity Maximization

-Regularization by Noise of Singular ODE's

Regularization of linear inverse problems with irregular noise using embedding operators

The Regularization Effects of Anisotropic Noise in Stochastic Gradient Descent.

Deep Active Learning with Noise Stability