Abstract:We initiate the study of nonsmooth optimization problems under bounded local subgradient variation, which postulates bounded difference between (sub)gradients in small local regions around points, in either average or maximum sense. The resulting class of objective functions encapsulates the classes of objective functions traditionally studied in optimization, which are defined based on either Lipschitz continuity of the objective or Hölder/Lipschitz continuity of its gradient. Further, the defined class contains functions that are neither Lipschitz continuous nor have a Hölder continuous gradient. When restricted to the traditional classes of optimization problems, the parameters defining the studied classes lead to more fine-grained complexity bounds, recovering traditional oracle complexity bounds in the worst case but generally leading to lower oracle complexity for functions that are not ``worst case.'' Some highlights of our results are that: (i) it is possible to obtain complexity results for both convex and nonconvex problems with the (local or global) Lipschitz constant being replaced by a constant of local subgradient variation and (ii) mean width of the subdifferential set around the optima plays a role in the complexity of nonsmooth optimization, particularly in parallel settings. A consequence of (ii) is that for any error parameter $\epsilon > 0$, parallel oracle complexity of nonsmooth Lipschitz convex optimization is lower than its sequential oracle complexity by a factor $\tilde{\Omega}\big(\frac{1}{\epsilon}\big)$ whenever the objective function is piecewise linear with polynomially many pieces in the input size. This is particularly surprising as existing parallel complexity lower bounds are based on such classes of functions. The seeming contradiction is resolved by considering the region in which the algorithm is allowed to query the objective.

Lower Bounds on the Noiseless Worst-Case Complexity of Efficient Global Optimization

Lower Bounds on the Oracle Complexity of Nonsmooth Convex Optimization via Information Theory

On Lower Complexity Bounds for Large-Scale Smooth Convex Optimization

Complexity Lower Bounds for Nonconvex-Strongly-Concave Min-Max Optimization.

Lower and Upper Bounds on the Generalization of Stochastic Exponentially Concave Optimization.

Improved Iteration Complexity in Black-Box Optimization Problems under Higher Order Smoothness Function Condition

High Probability Complexity Bounds for Non-Smooth Stochastic Optimization with Heavy-Tailed Noise

High-Probability Complexity Bounds for Non-smooth Stochastic Convex Optimization with Heavy-Tailed Noise

Beyond Worst-Case Analysis in Stochastic Approximation: Moment Estimation Improves Instance Complexity.

Acceleration with a Ball Optimization Oracle

Optimization on a Finer Scale: Bounded Local Subgradient Variation Perspective

High Probability Complexity Bounds for Adaptive Step Search Based on Stochastic Oracles

On the Optimal Bounds for Noisy Computing

On the Oracle Complexity of Higher-Order Smooth Non-Convex Finite-Sum Optimization

On the Hardness of Meaningful Local Guarantees in Nonsmooth Nonconvex Optimization

Random Gradient-Free Minimization of Convex Functions

Worst Case Complexity Bounds for Linesearch-Type Derivative-Free Algorithms

Efficient Submodular Optimization under Noise: Local Search is Robust

Stochastic Nonsmooth Convex Optimization with Heavy-Tailed Noises: High-Probability Bound, In-Expectation Rate and Initial Distance Adaptation

Complexity of Minimizing Projected-Gradient-Dominated Functions with Stochastic First-order Oracles

Zeroth-order Random Subspace Algorithm for Non-smooth Convex Optimization