Abstract:This work studies minimization problems with zero-order noisy oracle information under the assumption that the objective function is highly smooth and possibly satisfies additional properties. We consider two kinds of zero-order projected gradient descent algorithms, which differ in the form of the gradient estimator. The first algorithm uses a gradient estimator based on randomization over the $\ell_2$ sphere due to Bach and Perchet (2016). We present an improved analysis of this algorithm on the class of highly smooth and strongly convex functions studied in the prior work, and we derive rates of convergence for two more general classes of non-convex functions. Namely, we consider highly smooth functions satisfying the Polyak-Łojasiewicz condition and the class of highly smooth functions with no additional property. The second algorithm is based on randomization over the $\ell_1$ sphere, and it extends to the highly smooth setting the algorithm that was recently proposed for Lipschitz convex functions in Akhavan et al. (2022). We show that, in the case of noiseless oracle, this novel algorithm enjoys better bounds on bias and variance than the $\ell_2$ randomization and the commonly used Gaussian randomization algorithms, while in the noisy case both $\ell_1$ and $\ell_2$ algorithms benefit from similar improved theoretical guarantees. The improvements are achieved thanks to a new proof techniques based on Poincaré type inequalities for uniform distributions on the $\ell_1$ or $\ell_2$ spheres. The results are established under weak (almost adversarial) assumptions on the noise. Moreover, we provide minimax lower bounds proving optimality or near optimality of the obtained upper bounds in several cases.

Minimisation of Polyak-Łojasewicz Functions Using Random Zeroth-Order Oracles

On the Lower Bound of Minimizing Polyak-Łojasiewicz Functions

Highly Smooth Zeroth-Order Methods for Solving Optimization Problems under the PL Condition

Smooth Convex Optimization using Sub-Zeroth-Order Oracles

Uniformly Optimal and Parameter-free First-order Methods for Convex and Function-constrained Optimization

On the Complexity of Finite-Sum Smooth Optimization under the Polyak-Łojasiewicz Condition

Zeroth-order Random Subspace Algorithm for Non-smooth Convex Optimization

Gradient-free optimization of highly smooth functions: improved analysis and a new algorithm

A Sequential Quadratic Programming Method with High Probability Complexity Bounds for Nonlinear Equality Constrained Stochastic Optimization

Oracle Complexity Separation in Convex Optimization

On the oracle complexity of smooth strongly convex minimization

First- and Second-Order High Probability Complexity Bounds for Trust-Region Methods with Noisy Oracles

Nesterov's method of dichotomy via Order Oracle: The problem of optimizing a two-variable function on a square

Complexity of Minimizing Projected-Gradient-Dominated Functions with Stochastic First-order Oracles

On the complexity of quasiconvex integer minimization problem

Ruppert-Polyak averaging for Stochastic Order Oracle

On the complexity of finding a local minimizer of a quadratic function over a polytope

First-order Methods for Affinely Constrained Composite Non-convex Non-smooth Problems: Lower Complexity Bound and Near-optimal Methods

Quantized Distributed Nonconvex Optimization Algorithms with Linear Convergence under the Polyak--$Ł$ojasiewicz Condition

High Probability Complexity Bounds for Adaptive Step Search Based on Stochastic Oracles

Lp quasi-norm minimization: algorithm and applications