Abstract:In this paper we provide a novel analytical perspective on the theoretical understanding of gradient-based learning algorithms by interpreting consensus-based optimization (CBO), a recently proposed multi-particle derivative-free optimization method, as a stochastic relaxation of gradient descent. Remarkably, we observe that through communication of the particles, CBO exhibits a stochastic gradient descent (SGD)-like behavior despite solely relying on evaluations of the objective function. The fundamental value of such link between CBO and SGD lies in the fact that CBO is provably globally convergent to global minimizers for ample classes of nonsmooth and nonconvex objective functions, hence, on the one side, offering a novel explanation for the success of stochastic relaxations of gradient descent. On the other side, contrary to the conventional wisdom for which zero-order methods ought to be inefficient or not to possess generalization abilities, our results unveil an intrinsic gradient descent nature of such heuristics. This viewpoint furthermore complements previous insights into the working principles of CBO, which describe the dynamics in the mean-field limit through a nonlinear nonlocal partial differential equation that allows to alleviate complexities of the nonconvex function landscape. Our proofs leverage a completely nonsmooth analysis, which combines a novel quantitative version of the Laplace principle (log-sum-exp trick) and the minimizing movement scheme (proximal iteration). In doing so, we furnish useful and precise insights that explain how stochastic perturbations of gradient descent overcome energy barriers and reach deep levels of nonconvex functions. Instructive numerical illustrations support the provided theoretical insights.

An overview of gradient descent optimization algorithms

Gradient Descent, Stochastic Optimization, and Other Tales

Gradient Descent: The Ultimate Optimizer

A Comprehensive Study on Optimization Strategies for Gradient Descent In Deep Learning

Enhancing Deep Learning with Optimized Gradient Descent: Bridging Numerical Methods and Neural Network Training

Gradient is All You Need?

On nonconvex optimization for machine learning: Gradients, stochasticity, and saddle points

Gradient Descent based Optimization Algorithms for Deep Learning Models Training

A gentle introduction to gradient-based optimization and variational inequalities for machine learning

Optimizing Optimizers: Regret-optimal gradient descent algorithms

Optimal Adaptive and Accelerated Stochastic Gradient Descent

Optimization Methods in Deep Learning: A Comprehensive Overview

Gradients are Not All You Need

A gradient descent akin method for constrained optimization: algorithms and applications

Automatic Gradient Descent: Deep Learning without Hyperparameters

Reparametrizing gradient descent

A Primer on Coordinate Descent Algorithms

Optimization for deep learning: theory and algorithms

Stochastic Gradient Descent in the Viewpoint of Graduated Optimization

Convergence and Trade-Offs in Riemannian Gradient Descent and Riemannian Proximal Point

Random Function Descent