Abstract:Zeroth-order (ZO) optimization has become a popular technique for solving machine learning (ML) problems when first-order (FO) information is difficult or impossible to obtain. However, the scalability of ZO optimization remains an open problem: Its use has primarily been limited to relatively small-scale ML problems, such as sample-wise adversarial attack generation. To our best knowledge, no prior work has demonstrated the effectiveness of ZO optimization in training deep neural networks (DNNs) without a significant decrease in performance. To overcome this roadblock, we develop DeepZero, a principled ZO deep learning (DL) framework that can scale ZO optimization to DNN training from scratch through three primary innovations. First, we demonstrate the advantages of coordinatewise gradient estimation (CGE) over randomized vector-wise gradient estimation in training accuracy and computational efficiency. Second, we propose a sparsityinduced ZO training protocol that extends the model pruning methodology using only finite differences to explore and exploit the sparse DL prior in CGE. Third, we develop the methods of feature reuse and forward parallelization to advance the practical implementations of ZO training. Our extensive experiments show that DeepZero achieves state-of-the-art (SOTA) accuracy on ResNet-20 trained on CIFAR-10, approaching FO training performance for the first time. Furthermore, we show the practical utility of DeepZero in applications of certified adversarial defense and DL-based partial differential equation error correction, achieving 10-20% improvement over SOTA. We believe our results will inspire future research on scalable ZO optimization and contribute to advancing DL with black box. Codes are available at <a class="link-external link-https" href="https://github.com/OPTML-Group/DeepZero" rel="external noopener nofollow">this https URL</a>.

ZOOpt: a Toolbox for Derivative-Free Optimization.

A Simultaneous Strategy For Dynamic Optimization Based On Symbolic Derivation

Design and Implementation of Large-scale Nonlinear Optimization Software UniOptima

OpenBox: A Python Toolkit for Generalized Black-box Optimization

Unifying nonlinearly constrained nonconvex optimization

Zebra Optimization Algorithm: A New Bio-inspired Optimization Algorithm for Solving Optimization Algorithm

Derivative-free tree optimization for complex systems

Quadratic Interpolation Optimization (QIO): A new optimization algorithm based on generalized quadratic interpolation and its applications to real-world engineering problems

DeepZero: Scaling up Zeroth-Order Optimization for Deep Model Training

Optimization Condition and Algorithm of Optimization with Convertible Nonconvex Function

Curvature-Aware Derivative-Free Optimization

Federated Zeroth-Order Optimization using Trajectory-Informed Surrogate Gradients

OpenBox: A Generalized Black-box Optimization Service

SOSOPT: A Toolbox for Polynomial Optimization

A derivative-free approach to partitioned optimization

Obtaining Lower Query Complexities Through Lightweight Zeroth-Order Proximal Gradient Algorithms

Stochastic Zeroth-order Optimization Via Variance Reduction Method.

A Hessian-based zeroing neurodynamic approach for quaternion-variable time-varying constrained optimization problems

TorchOpt: an Efficient Library for Differentiable Optimization.

Safe Zeroth-Order Optimization Using Quadratic Local Approximations

Black-box Optimization Algorithms for Regularized Least-squares Problems