Abstract:We present a novel gradient-free algorithm to solve a convex stochastic optimization problem, such as those encountered in medicine, physics, and machine learning (e.g., adversarial multi-armed bandit problem), where the objective function can only be computed through numerical simulation, either as the result of a real experiment or as feedback given by the function evaluations from an adversary. Thus we suppose that only a black-box access to the function values of the objective is available, possibly corrupted by adversarial noise: deterministic or stochastic. The noisy setup can arise naturally from modeling randomness within a simulation or by computer discretization, or when exact values of function are forbidden due to privacy issues, or when solving non-convex problems as convex ones with an inexact function oracle. By exploiting higher-order smoothness, fulfilled, e.g., in logistic regression, we improve the performance of zero-order methods developed under the assumption of classical smoothness (or having a Lipschitz gradient). The proposed algorithm enjoys optimal oracle complexity and is designed under an overparameterization setup, i.e., when the number of model parameters is much larger than the size of the training dataset. Overparametrized models fit to the training data perfectly while also having good generalization and outperforming underparameterized models on unseen data. We provide convergence guarantees for the proposed algorithm under both types of noise. Moreover, we estimate the maximum permissible adversarial noise level that maintains the desired accuracy in the Euclidean setup, and then we extend our results to a non-Euclidean setup. Our theoretical results are verified on the logistic regression problem.

On the global complexity of a derivative-free Levenberg-Marquardt algorithm via orthogonal spherical smoothing

A New Algorithm For Unconstrained Optimization Problem With The Form Of Sum Of Squares Minimization

Complexity bound of a Levenberg–Marquardt algorithm based on probabilistic Jacobian models

Scalable Subspace Methods for Derivative-Free Nonlinear Least-Squares Optimization

Global Convergence of a Stochastic Levenberg–Marquardt Algorithm Based on Trust Region

Q-fully quadratic modeling and its application in a random subspace derivative-free method

$Q$-fully Quadratic Modeling and its Application in a Random Subspace Derivative-free Method

A new approximate descent derivative-free algorithm for large-scale nonlinear symmetric equations

A fresh look at nonsmooth Levenberg–Marquardt methods with applications to bilevel optimization

On complexity constants of linear and quadratic models for derivative-free trust-region algorithms

A Single-Loop Stochastic Proximal Quasi-Newton Method for Large-Scale Nonsmooth Convex Optimization

Black-box Optimization Algorithms for Regularized Least-squares Problems

Riemannian Smoothing Gradient Type Algorithms]{Riemannian Smoothing Gradient Type Algorithms for Nonsmooth Optimization Problem on Compact Riemannian Submanifold Embedded in Euclidean Space

A Stochastic Semismooth Newton Method for Nonsmooth Nonconvex Optimization.

Stochastic Zeroth-order Optimization Via Variance Reduction Method.

Accelerated zero-order SGD under high-order smoothness and overparameterized regime

A Smoothing Stochastic Gradient Method for Composite Optimization

Zeroth-order Random Subspace Algorithm for Non-smooth Convex Optimization

An Efficient Modified Residual-Based Algorithm for Large Scale Symmetric Nonlinear Equations by Approximating Successive Iterated Gradients

Derivative-free Alternating Projection Algorithms for General Nonconvex-Concave Minimax Problems