Abstract:In this paper we provide a rigorous convergence analysis for the renowned particle swarm optimization method by using tools from stochastic calculus and the analysis of partial differential equations. Based on a continuous-time formulation of the particle dynamics as a system of stochastic differential equations, we establish convergence to a global minimizer of a possibly nonconvex and nonsmooth objective function in two steps. First, we prove consensus formation of an associated mean-field dynamics by analyzing the time-evolution of the variance of the particle distribution, which acts as Lyapunov function of the dynamics. We then show that this consensus is close to a global minimizer by employing the asymptotic Laplace principle and a tractability condition on the energy landscape of the objective function. These results allow for the usage of memory mechanisms, and hold for a rich class of objectives provided certain conditions of well-preparation of the hyperparameters and the initial datum. In a second step, at least for the case without memory effects, we provide a quantitative result about the mean-field approximation of particle swarm optimization, which specifies the convergence of the interacting particle system to the associated mean-field limit. Combining these two results allows for global convergence guarantees of the numerical particle swarm optimization method with provable polynomial complexity. To demonstrate the applicability of the method we propose an efficient and parallelizable implementation, which is tested in particular on a competitive and well-understood high-dimensional benchmark problem in machine learning.

Global convergence and asymptotic optimality of the heavy ball method for a class of non-convex optimization problems

A Robust Control Approach to Asymptotic Optimality of the Heavy Ball Method for Optimization of Quadratic Functions

Accelerated Over-Relaxation Heavy-Ball Methods with Provable Acceleration and Global Convergence

Non-Ergodic Convergence Analysis of Heavy-Ball Algorithms.

Heavy Ball Momentum for Non-Strongly Convex Optimization

Convergence analysis of a stochastic heavy-ball method for linear ill-posed problems

Polyak's Heavy Ball Method Achieves Accelerated Local Rate of Convergence under Polyak-Lojasiewicz Inequality

On the Convergence Analysis of Aggregated Heavy-Ball Method

Nonsmooth Nonconvex Stochastic Heavy Ball

Convergence and Uniqueness Properties of Gauss-Newton's Method

Extended Newton Methods for Multiobjective Optimization: Majorizing Function Technique and Convergence Analysis

On the Powerball Method for Optimization

A Globally Convergent Gradient Method with Momentum

Universal heavy-ball method for nonconvex optimization under Hölder continuous Hessians

On growth error bound conditions with an application to heavy ball method

Primitive Heavy-ball Dynamics Achieves $O(\varepsilon^{-7/4})$ Convergence for Nonconvex Optimization

Unified Convergence Analysis of Stochastic Momentum Methods for Convex and Non-convex Optimization

Global Convergence Analysis of a New Nonmonotone BFGS Algorithm on Convex Objective Functions

On the Global Convergence of Particle Swarm Optimization Methods

Accelerated Convergence of Stochastic Heavy Ball Method under Anisotropic Gradient Noise

A globalization of L-BFGS and the Barzilai-Borwein method for nonconvex unconstrained optimization