Abstract:We consider the minimization of a Lipschitz continuous and expectation-valued function defined as $f(\mathbf{x}) \triangleq \mathbb{E}[{\tilde f}(\mathbf{x}, \boldsymbol{\xi})]$, over a closed and convex set. Our focus lies on obtaining both asymptotics as well as rate and complexity guarantees for computing an approximate stationary point (in a Clarke sense) via zeroth-order schemes. We adopt a smoothing-based approach reliant on minimizing $f_{\eta}$ where $f_{\eta}(\mathbf{x}) = \mathbb{E}_{\mathbf{u}}[f(\mathbf{x}+\eta \mathbf{u})]$, $\mathbf{u}$ is a random variable defined on a unit sphere, and $\eta > 0$. It has been observed that a stationary point of the $\eta$-smoothed problem is a $2\eta$-stationary point for the original problem in the Clarke sense. In such a setting, we develop two sets of schemes with promising empirical behavior. (I) We develop a smoothing-enabled variance-reduced zeroth-order gradient framework (VRG-ZO) and make two sets of contributions for the sequence generated by the proposed zeroth-order gradient scheme. (a) The residual function of the smoothed problem tends to zero almost surely along the generated sequence, allowing for making guarantees for $\eta$-Clarke stationary solutions of the original problem; (b) To compute an $\mathbf{x}$ that ensures that the expected norm of the residual of the $\eta$-smoothed problem is within $\epsilon$ requires no greater than $O(\eta^{-1} \epsilon^{-2})$ projection steps and $ O\left(\eta^{-2} \epsilon^{-4}\right)$ function evaluations. (II) Our second scheme is a zeroth-order stochastic quasi-Newton scheme (VRSQN-ZO) reliant on a combination of randomized and Moreau smoothing; the corresponding iteration and sample complexities for this scheme are $ O\left(\eta^{-5}\epsilon^{-2}\right)$ and $ O\left(\eta^{-7}\epsilon^{-4}\right)$, respectively

Convergence Rates of Zeroth Order Gradient Descent for Łojasiewicz Functions

Convergence Rates of Stochastic Zeroth-order Gradient Descent for \L ojasiewicz Functions

Code and Data Repository for Convergence Rates of Zeroth Order Gradient Descent for Łojasiewicz Functions

Convergence Analysis of Gradient Algorithms on Riemannian Manifolds Without Curvature Constraints and Application to Riemannian Mass

Stochastic Zeroth order Descent with Structured Directions

Non-Uniform Smoothness for Gradient Descent

Optimizing $(L_0, L_1)$-Smooth Functions by Gradient Methods

Zero-Order Stochastic Conditional Gradient Sliding Method for Non-smooth Convex Optimization

Zero-Gradient-Sum Algorithms for Distributed Convex Optimization: The Continuous-Time Case

Linear Convergence of Adaptive Stochastic Gradient Descent

Inexact Riemannian Gradient Descent Method for Nonconvex Optimization

Improved Zeroth-Order Variance Reduced Algorithms and Analysis for Nonconvex Optimization

Zeroth-order Gradient and Quasi-Newton Methods for Nonsmooth Nonconvex Stochastic Optimization

Laplacian Smoothing Gradient Descent

Convergence of $\text{log}(1/ε)$ for Gradient-Based Algorithms in Zero-Sum Games without the Condition Number: A Smoothed Analysis

The Novel Adaptive Fractional Order Gradient Decent Algorithms Design via Robust Control

On the Lower Bound of Minimizing Polyak-Łojasiewicz Functions

On the Convergence of AdaGrad(Norm) on $\R^{d}$: Beyond Convexity, Non-Asymptotic Rate and Acceleration

Zeroth-Order Methods for Constrained Nonconvex Nonsmooth Stochastic Optimization

Universal Gradient Descent Ascent Method for Nonconvex-Nonconcave Minimax Optimization