Abstract:In this paper we provide a rigorous convergence analysis for the renowned particle swarm optimization method by using tools from stochastic calculus and the analysis of partial differential equations. Based on a continuous-time formulation of the particle dynamics as a system of stochastic differential equations, we establish convergence to a global minimizer of a possibly nonconvex and nonsmooth objective function in two steps. First, we prove consensus formation of an associated mean-field dynamics by analyzing the time-evolution of the variance of the particle distribution, which acts as Lyapunov function of the dynamics. We then show that this consensus is close to a global minimizer by employing the asymptotic Laplace principle and a tractability condition on the energy landscape of the objective function. These results allow for the usage of memory mechanisms, and hold for a rich class of objectives provided certain conditions of well-preparation of the hyperparameters and the initial datum. In a second step, at least for the case without memory effects, we provide a quantitative result about the mean-field approximation of particle swarm optimization, which specifies the convergence of the interacting particle system to the associated mean-field limit. Combining these two results allows for global convergence guarantees of the numerical particle swarm optimization method with provable polynomial complexity. To demonstrate the applicability of the method we propose an efficient and parallelizable implementation, which is tested in particular on a competitive and well-understood high-dimensional benchmark problem in machine learning.

On the Minimum Convergence Factor of a Class of GSOR-like Methods for Augmented Systems

On convergence and semi-convergence of SSOR-like methods for augmented linear systems

On a New SSOR-Like Method with Four Parameters for the Augmented Systems

Optimal Parameters of GSOR-like Methods for Solving the Augmented Linear Systems

The Convergence Rate and Necessary-and-Sufficient Condition for the Consistency of Isogeometric Collocation Method

Convergence Behavior of Gauss-Newton's Method and Extensions of the Smale Point Estimate Theory.

Convergence of A Generalized Usor Iterative Method for Augmented Systems

Convergence of a generalized MSSOR method for augmented systems

Extended Newton Methods for Multiobjective Optimization: Majorizing Function Technique and Convergence Analysis

Modified Generalised Successive Over-Relaxation Method for Augmented Linear Systems

On generalized successive overrelaxation methods for augmented linear systems

The semi-convergence of generalized SSOR method for singular augmented systems

A Modified SSOR Iterative Method for Augmented Systems

Global Convergence of Langevin Dynamics Based Algorithms for Nonconvex Optimization

Two efficient gradient methods with approximately optimal stepsizes based on regularization models for unconstrained optimization

A Unified Convergence Analysis of a Second-Order Method of Multipliers for Nonlinear Conic Programming

Optimal maximum norm estimates for virtual element methods

Modified SSOR-like Method for Augmented Systems

An efficient inertial subspace minimization CG algorithm with convergence rate analysis for constrained nonlinear monotone equations

On the Global Convergence of Particle Swarm Optimization Methods