Abstract:Zeroth-order optimization is a fundamental research topic that has been a focus of various learning tasks, such as black-box adversarial attacks, bandits, and reinforcement learning. However, in theory, most complexity results assert a linear dependency on the dimension of optimization variable, which implies paralyzations of zeroth-order algorithms for high-dimensional problems and cannot explain their effectiveness in practice. In this paper, we present a novel zeroth-order optimization theory characterized by complexities that exhibit weak dependencies on dimensionality. The key contribution lies in the introduction of a new factor, denoted as $\mathrm{ED}_{\alpha}=\sup_{x\in \mathbb{R}^d}\sum_{i=1}^d\sigma_i^\alpha(\nabla^2 f(x))$ ($\alpha>0$, $\sigma_i(\cdot)$ is the $i$-th singular value in non-increasing order), which effectively functions as a measure of dimensionality. The algorithms we propose demonstrate significantly reduced complexities when measured in terms of the factor $\mathrm{ED}_{\alpha}$. Specifically, we first study a well-known zeroth-order algorithm from Nesterov and Spokoiny (2017) on quadratic objectives and show a complexity of $\mathcal{O}\left(\frac{\mathrm{ED}_1}{\sigma_d}\log(1/\epsilon)\right)$ for the strongly convex setting. Furthermore, we introduce novel algorithms that leverages the Heavy-ball mechanism. Our proposed algorithm exhibits a complexity of $\mathcal{O}\left(\frac{\mathrm{ED}_{1/2}}{\sqrt{\sigma_d}}\cdot\log{\frac{L}{\mu}}\cdot\log(1/\epsilon)\right)$. We further expand the scope of the method to encompass generic smooth optimization problems under an additional Hessian-smooth condition. The resultant algorithms demonstrate remarkable complexities which improve by an order in $d$ under appropriate conditions. Our analysis lays the foundation for zeroth-order optimization methods for smooth functions within high-dimensional settings.

Zeroth-order Random Subspace Algorithm for Non-smooth Convex Optimization

Stochastic Zeroth-order Optimization Via Variance Reduction Method.

Randomized subspace gradient method for constrained optimization

A Dimension-Insensitive Algorithm for Stochastic Zeroth-Order Optimization

Zeroth-Order Methods for Constrained Nonconvex Nonsmooth Stochastic Optimization

Zeroth-order Gradient and Quasi-Newton Methods for Nonsmooth Nonconvex Stochastic Optimization

Smooth Convex Optimization using Sub-Zeroth-Order Oracles

Derivative-Free Alternating Projection Algorithms for General Nonconvex-Concave Minimax Problems

An Algorithm with Optimal Dimension-Dependence for Zero-Order Nonsmooth Nonconvex Stochastic Optimization

Zeroth-order algorithms for stochastic distributed nonconvex optimization

Randomised subspace methods for non-convex optimization, with applications to nonlinear least-squares

Gradient-free optimization of highly smooth functions: improved analysis and a new algorithm

Zeroth-Order Methods for Nonconvex Stochastic Problems with Decision-Dependent Distributions

Zeroth-order Optimization with Weak Dimension Dependency

Accelerated zero-order SGD under high-order smoothness and overparameterized regime

Randomized Subspace Derivative-Free Optimization with Quadratic Models and Second-Order Convergence

Distributed Zeroth-Order Optimization: Convergence Rates That Match Centralized Counterpart

Obtaining Lower Query Complexities Through Lightweight Zeroth-Order Proximal Gradient Algorithms

Improved Zeroth-Order Variance Reduced Algorithms and Analysis for Nonconvex Optimization

Subspace Quasi-Newton Method with Gradient Approximation