Abstract:Recently, several studies consider the stochastic optimization problem but in a heavy-tailed noise regime, i.e., the difference between the stochastic gradient and the true gradient is assumed to have a finite $p$-th moment (say being upper bounded by $\sigma^{p}$ for some $\sigma\geq0$) where $p\in(1,2]$, which not only generalizes the traditional finite variance assumption ($p=2$) but also has been observed in practice for several different tasks. Under this challenging assumption, lots of new progress has been made for either convex or nonconvex problems, however, most of which only consider smooth objectives. In contrast, people have not fully explored and well understood this problem when functions are nonsmooth. This paper aims to fill this crucial gap by providing a comprehensive analysis of stochastic nonsmooth convex optimization with heavy-tailed noises. We revisit a simple clipping-based algorithm, whereas, which is only proved to converge in expectation but under the additional strong convexity assumption. Under appropriate choices of parameters, for both convex and strongly convex functions, we not only establish the first high-probability rates but also give refined in-expectation bounds compared with existing works. Remarkably, all of our results are optimal (or nearly optimal up to logarithmic factors) with respect to the time horizon $T$ even when $T$ is unknown in advance. Additionally, we show how to make the algorithm parameter-free with respect to $\sigma$, in other words, the algorithm can still guarantee convergence without any prior knowledge of $\sigma$. Furthermore, an initial distance adaptive convergence rate is provided if $\sigma$ is assumed to be known.

High Probability Complexity Bounds for Adaptive Step Search Based on Stochastic Oracles

Sample Complexity Analysis for Adaptive Optimization Algorithms with Stochastic Oracles

High Probability Complexity Bounds for Non-Smooth Stochastic Optimization with Heavy-Tailed Noise

High-Probability Complexity Bounds for Non-smooth Stochastic Convex Optimization with Heavy-Tailed Noise

Universality of AdaGrad Stepsizes for Stochastic Optimization: Inexact Oracle, Acceleration and Variance Reduction

First- and Second-Order High Probability Complexity Bounds for Trust-Region Methods with Noisy Oracles

Stochastic Optimization Algorithms for Problems with Controllable Biased Oracles

A Sequential Quadratic Programming Method with High Probability Complexity Bounds for Nonlinear Equality Constrained Stochastic Optimization

Complexity of Minimizing Projected-Gradient-Dominated Functions with Stochastic First-order Oracles

On the Complexity of First-Order Methods in Stochastic Bilevel Optimization

On the Oracle Complexity of Higher-Order Smooth Non-Convex Finite-Sum Optimization

Optimal inexactness schedules for Tunable Oracle based Methods

The Minimax Complexity of Distributed Optimization

High-Probability Bounds for Stochastic Optimization and Variational Inequalities: the Case of Unbounded Variance

Optimal inexactness schedules for tunable oracle-based methods

A stochastic extra-step quasi-Newton method for nonsmooth nonconvex optimization

Beyond Worst-Case Analysis in Stochastic Approximation: Moment Estimation Improves Instance Complexity.

Stochastic Nonsmooth Convex Optimization with Heavy-Tailed Noises: High-Probability Bound, In-Expectation Rate and Initial Distance Adaptation

Stochastic Optimization for Nonconvex Problem with Inexact Hessian Matrix, Gradient, and Function

Nonasymptotic Analysis of Accelerated Methods With Inexact Oracle Under Absolute Error Bound

Gradient Methods with Dynamic Inexact Oracles