Abstract:Bi-level optimization problems, where one wishes to find the global minimizer of an upper-level objective function over the globally optimal solution set of a lower-level objective, arise in a variety of scenarios throughout science and engineering, machine learning, and artificial intelligence. In this paper, we propose and investigate, analytically and experimentally, consensus-based bi-level optimization (CB$^2$O), a multi-particle metaheuristic derivative-free optimization method designed to solve bi-level optimization problems when both objectives may be nonconvex. Our method leverages within the computation of the consensus point a carefully designed particle selection principle implemented through a suitable choice of a quantile on the level of the lower-level objective, together with a Laplace principle-type approximation w.r.t. the upper-level objective function, to ensure that the bi-level optimization problem is solved in an intrinsic manner. We give an existence proof of solutions to a corresponding mean-field dynamics, for which we first establish the stability of our consensus point w.r.t. a combination of Wasserstein and $L^2$ perturbations, and consecutively resort to PDE considerations extending the classical Picard iteration to construct a solution. For such solution, we provide a global convergence analysis in mean-field law showing that the solution of the associated nonlinear nonlocal Fokker-Planck equation converges exponentially fast to the unique solution of the bi-level optimization problem provided suitable choices of the hyperparameters. The practicability and efficiency of our CB$^2$O algorithm is demonstrated through extensive numerical experiments in the settings of constrained global optimization, sparse representation learning, and robust (clustered) federated learning.

Consensus-based optimisation with truncated noise

Constrained Consensus-Based Optimization and Numerical Heuristics for the Few Particle Regime

Consensus-Based Optimization Methods Converge Globally

A PDE Framework of Consensus-Based Optimization for Objectives with Multiple Global Minimizers

A discrete Consensus-Based Global Optimization Method with Noisy Objective Function

Consensus under Bounded Noise in Discrete Network Systems: an Algorithm with Fast Convergence and High Accuracy.

Fast and robust consensus-based optimization via optimal feedback control

Leveraging Memory Effects and Gradient Information in Consensus-Based Optimization: On Global Convergence in Mean-Field Law

Convergence of Anisotropic Consensus-Based Optimization in Mean-Field Law

Convergence analysis of the discrete consensus-based optimization algorithm with random batch interactions and heterogeneous noises

A multiscale Consensus-Based algorithm for multi-level optimization

Discrete Consensus-Based Optimization

Consensus-Based Optimization for Saddle Point Problems

Convergence and error estimates for time-discrete consensus-based optimization algorithms

Zero-Inertia Limit: from Particle Swarm Optimization to Consensus-Based Optimization.

Uniform-in-time mean-field limit estimate for the Consensus-Based Optimization

CB$^2$O: Consensus-Based Bi-Level Optimization

Gradient is All You Need?

Tradeoffs between convergence rate and noise amplification for momentum-based accelerated optimization algorithms

Consensus based optimization with memory effects: random selection and applications