Abstract:In this paper we provide a rigorous convergence analysis for the renowned particle swarm optimization method by using tools from stochastic calculus and the analysis of partial differential equations. Based on a continuous-time formulation of the particle dynamics as a system of stochastic differential equations, we establish convergence to a global minimizer of a possibly nonconvex and nonsmooth objective function in two steps. First, we prove consensus formation of an associated mean-field dynamics by analyzing the time-evolution of the variance of the particle distribution, which acts as Lyapunov function of the dynamics. We then show that this consensus is close to a global minimizer by employing the asymptotic Laplace principle and a tractability condition on the energy landscape of the objective function. These results allow for the usage of memory mechanisms, and hold for a rich class of objectives provided certain conditions of well-preparation of the hyperparameters and the initial datum. In a second step, at least for the case without memory effects, we provide a quantitative result about the mean-field approximation of particle swarm optimization, which specifies the convergence of the interacting particle system to the associated mean-field limit. Combining these two results allows for global convergence guarantees of the numerical particle swarm optimization method with provable polynomial complexity. To demonstrate the applicability of the method we propose an efficient and parallelizable implementation, which is tested in particular on a competitive and well-understood high-dimensional benchmark problem in machine learning.

Practical Convergence of Distributed Stochastic Iterative Processes

On the Convergence of the Accelerated Riccati Iteration Method.

Convergence Speed Analysis and Optimization for Distributed Control of Virtual Power Plant

On Convergence to Stochastic Integrals

Necessary and sufficient condition for mean-square convergence of linear distributed iterative stochastic processes

Convergence in High Probability of Distributed Stochastic Gradient Descent Algorithms

On Decomposition and Convergence of Distributed Optimization Algorithms

On the Convergence of Parallel Asynchronous Block-Iterative Computations

On Inhomogeneous Infinite Products of Stochastic Matrices and Applications

Linear convergence of a primal-dual algorithm for distributed interval optimization

Convergence Theorems for Stochastic Impulsive Systems with Application to Discrete-Time Stochastic Feedback Control

On Inhomogeneous Infinite Products of Stochastic Matrices and Their Applications

Linear Convergence of Primal-Dual Gradient Methods and their Performance in Distributed Optimization

Convergence of Asynchronous Distributed Gradient Methods over Stochastic Networks

Distributed Constrained Optimization with Linear Convergence Rate

On the Global Convergence of Particle Swarm Optimization Methods

Stability and Convergence Rate Analysis of Continuous-Time Delay-Difference Systems with Both Point and Distributed Delays

Convergence of the Semi-implicit Euler Numerical Solutions of Stochastic Jump-diffusion Equations with Distributed Memory

Disquisition on convergence, stability, and data dependence for a new fast iterative process

A Distributed Control Law with Guaranteed Convergence Rate for Identically Coupled Linear Systems

On the convergence of the Gaver-Stehfest algorithm