Abstract:This work studies minimization problems with zero-order noisy oracle information under the assumption that the objective function is highly smooth and possibly satisfies additional properties. We consider two kinds of zero-order projected gradient descent algorithms, which differ in the form of the gradient estimator. The first algorithm uses a gradient estimator based on randomization over the $\ell_2$ sphere due to Bach and Perchet (2016). We present an improved analysis of this algorithm on the class of highly smooth and strongly convex functions studied in the prior work, and we derive rates of convergence for two more general classes of non-convex functions. Namely, we consider highly smooth functions satisfying the Polyak-Łojasiewicz condition and the class of highly smooth functions with no additional property. The second algorithm is based on randomization over the $\ell_1$ sphere, and it extends to the highly smooth setting the algorithm that was recently proposed for Lipschitz convex functions in Akhavan et al. (2022). We show that, in the case of noiseless oracle, this novel algorithm enjoys better bounds on bias and variance than the $\ell_2$ randomization and the commonly used Gaussian randomization algorithms, while in the noisy case both $\ell_1$ and $\ell_2$ algorithms benefit from similar improved theoretical guarantees. The improvements are achieved thanks to a new proof techniques based on Poincaré type inequalities for uniform distributions on the $\ell_1$ or $\ell_2$ spheres. The results are established under weak (almost adversarial) assumptions on the noise. Moreover, we provide minimax lower bounds proving optimality or near optimality of the obtained upper bounds in several cases.

Dynamic Anisotropic Smoothing for Noisy Derivative-Free Optimization

Fast inertial dynamic algorithm with smoothing method for nonsmooth convex optimization

A dynamic gradient approach to Pareto optimization with nonsmooth convex objective functions

Gaussian smoothing gradient descent for minimizing functions (GSmoothGD)

Laplacian Smoothing Gradient Descent

Constrained Global Optimization by Smoothing

Exploiting Negative Curvature in Conjunction with Adaptive Sampling: Theoretical Results and a Practical Algorithm

An Approximate Newton Smoothing Method for Shape Optimization

Edge-Aware Volume Smoothing Using L0 Gradient Minimization

A Gradient Smoothed Functional Algorithm with Truncated Cauchy Random Perturbations for Stochastic Optimization

Diagonalisation SGD: Fast & Convergent SGD for Non-Differentiable Models via Reparameterisation and Smoothing

Generalizing Stochastic Smoothing for Differentiation and Gradient Estimation

Using Stochastic Gradient Descent to Smooth Nonconvex Functions: Analysis of Implicit Graduated Optimization

A feasible smoothing accelerated projected gradient method for nonsmooth convex optimization

Adaptive State-Dependent Diffusion for Derivative-Free Optimization

Convex and Non-convex Optimization Under Generalized Smoothness

Gradient-free optimization of highly smooth functions: improved analysis and a new algorithm

A variable smoothing for Nonconvexly constrained nonsmooth optimization with application to sparse spectral clustering

Anisotropic Proximal Gradient

Nonsmooth Nonconvex Stochastic Heavy Ball

Using second-order information in gradient sampling methods for nonsmooth optimization