Abstract:We consider the minimization of a Lipschitz continuous and expectation-valued function defined as $f(\mathbf{x}) \triangleq \mathbb{E}[{\tilde f}(\mathbf{x}, \boldsymbol{\xi})]$, over a closed and convex set. Our focus lies on obtaining both asymptotics as well as rate and complexity guarantees for computing an approximate stationary point (in a Clarke sense) via zeroth-order schemes. We adopt a smoothing-based approach reliant on minimizing $f_{\eta}$ where $f_{\eta}(\mathbf{x}) = \mathbb{E}_{\mathbf{u}}[f(\mathbf{x}+\eta \mathbf{u})]$, $\mathbf{u}$ is a random variable defined on a unit sphere, and $\eta > 0$. It has been observed that a stationary point of the $\eta$-smoothed problem is a $2\eta$-stationary point for the original problem in the Clarke sense. In such a setting, we develop two sets of schemes with promising empirical behavior. (I) We develop a smoothing-enabled variance-reduced zeroth-order gradient framework (VRG-ZO) and make two sets of contributions for the sequence generated by the proposed zeroth-order gradient scheme. (a) The residual function of the smoothed problem tends to zero almost surely along the generated sequence, allowing for making guarantees for $\eta$-Clarke stationary solutions of the original problem; (b) To compute an $\mathbf{x}$ that ensures that the expected norm of the residual of the $\eta$-smoothed problem is within $\epsilon$ requires no greater than $O(\eta^{-1} \epsilon^{-2})$ projection steps and $ O\left(\eta^{-2} \epsilon^{-4}\right)$ function evaluations. (II) Our second scheme is a zeroth-order stochastic quasi-Newton scheme (VRSQN-ZO) reliant on a combination of randomized and Moreau smoothing; the corresponding iteration and sample complexities for this scheme are $ O\left(\eta^{-5}\epsilon^{-2}\right)$ and $ O\left(\eta^{-7}\epsilon^{-4}\right)$, respectively

A Riemannian Gradient Sampling Algorithm for Nonsmooth Optimization on Manifolds

Convergence Analysis of Gradient Algorithms on Riemannian Manifolds Without Curvature Constraints and Application to Riemannian Mass

Riemannian Smoothing Gradient Type Algorithms]{Riemannian Smoothing Gradient Type Algorithms for Nonsmooth Optimization Problem on Compact Riemannian Submanifold Embedded in Euclidean Space

Riemannian stochastic optimization methods avoid strict saddle points

Randomized Submanifold Subgradient Method for Optimization over Stiefel Manifolds

Riemannian Stochastic Proximal Gradient Methods for Nonsmooth Optimization over the Stiefel Manifold

A descent subgradient method using Mifflin's line search for nonsmooth nonconvex optimization

Learning-Rate-Free Stochastic Optimization over Riemannian Manifolds

Inexact Riemannian Gradient Descent Method for Nonconvex Optimization

Subgradient sampling for nonsmooth nonconvex minimization

Convergence of Inexact Steepest Descent Algorithm for Multiobjective Optimizations on Riemannian Manifolds Without Curvature Constraints

Analyzing the speed of convergence in nonsmooth optimization via the Goldstein subdifferential with application to descent methods

A note on the convergence of deterministic gradient sampling in nonsmooth optimization

Convergence and Trade-Offs in Riemannian Gradient Descent and Riemannian Proximal Point

A descent subgradient method using Mifflin line search for nonsmooth nonconvex optimization

A Decentralized Proximal Gradient Tracking Algorithm for Composite Optimization on Riemannian Manifolds

Gauss–Newton Method for Convex Composite Optimizations on Riemannian Manifolds

An Incremental Subgradient Method on Riemannian Manifolds

Zeroth-order Gradient and Quasi-Newton Methods for Nonsmooth Nonconvex Stochastic Optimization

Riemannian stochastic fixed point optimization algorithm

Gradient descent with a general cost