Abstract:In this paper we provide a rigorous convergence analysis for the renowned particle swarm optimization method by using tools from stochastic calculus and the analysis of partial differential equations. Based on a continuous-time formulation of the particle dynamics as a system of stochastic differential equations, we establish convergence to a global minimizer of a possibly nonconvex and nonsmooth objective function in two steps. First, we prove consensus formation of an associated mean-field dynamics by analyzing the time-evolution of the variance of the particle distribution, which acts as Lyapunov function of the dynamics. We then show that this consensus is close to a global minimizer by employing the asymptotic Laplace principle and a tractability condition on the energy landscape of the objective function. These results allow for the usage of memory mechanisms, and hold for a rich class of objectives provided certain conditions of well-preparation of the hyperparameters and the initial datum. In a second step, at least for the case without memory effects, we provide a quantitative result about the mean-field approximation of particle swarm optimization, which specifies the convergence of the interacting particle system to the associated mean-field limit. Combining these two results allows for global convergence guarantees of the numerical particle swarm optimization method with provable polynomial complexity. To demonstrate the applicability of the method we propose an efficient and parallelizable implementation, which is tested in particular on a competitive and well-understood high-dimensional benchmark problem in machine learning.

A Modified HS Conjugate Gradient Method and Its Global Convergence

The convergence of conjugate gradient method with nonmonotone line search

An improved Dai‐Liao‐style hybrid conjugate gradient‐based method for solving unconstrained nonconvex optimization and extension to constrained nonlinear monotone equations

Guaranteed Descent Conjugate Gradient Methods with Modified Secant Condition

Global Convergence of the Polak-Ribière and Hestenes-Stiefel Conjugate Gradient Methods for the Unconstrained Nonlinear Optimization

Convergence of the Descent Nonlinear Conjugate Gradient Methods

The global convergence of the BFGS method with a modified WWP line search for nonconvex functions

A convergent hybrid three-term conjugate gradient method with sufficient descent property for unconstrained optimization

Non-asymptotic Global Convergence Rates of BFGS with Exact Line Search

A New Three-Term Conjugate Gradient Algorithm with Modified Gradient-Differences for Solving Unconstrained Optimization Problems

Global convergence of Riemannian line search methods with a Zhang-Hager-type condition

A Modified Nonlinear Conjugate Gradient Algorithm for Large-Scale Nonsmooth Convex Optimization

Modified Line Search Sequential Quadratic Methods for Equality-Constrained Optimization with Unified Global and Local Convergence Guarantees

Alternative extension of the Hager–Zhang conjugate gradient method for vector optimization

A modified Polak-Ribiere-Polyak type conjugate gradient method with two stepsize strategies for vector optimization

On the extension of the Hager–Zhang conjugate gradient method for vector optimization

On Convergence of the Gauss-Newton Method for Convex Composite Optimization

Two modified conjugate gradient methods for unconstrained optimization

Extended Newton Methods for Multiobjective Optimization: Majorizing Function Technique and Convergence Analysis

A decent three term conjugate gradient method with global convergence properties for large scale unconstrained optimization problems

On the Global Convergence of Particle Swarm Optimization Methods