Abstract:Bi-level optimization problems, where one wishes to find the global minimizer of an upper-level objective function over the globally optimal solution set of a lower-level objective, arise in a variety of scenarios throughout science and engineering, machine learning, and artificial intelligence. In this paper, we propose and investigate, analytically and experimentally, consensus-based bi-level optimization (CB$^2$O), a multi-particle metaheuristic derivative-free optimization method designed to solve bi-level optimization problems when both objectives may be nonconvex. Our method leverages within the computation of the consensus point a carefully designed particle selection principle implemented through a suitable choice of a quantile on the level of the lower-level objective, together with a Laplace principle-type approximation w.r.t. the upper-level objective function, to ensure that the bi-level optimization problem is solved in an intrinsic manner. We give an existence proof of solutions to a corresponding mean-field dynamics, for which we first establish the stability of our consensus point w.r.t. a combination of Wasserstein and $L^2$ perturbations, and consecutively resort to PDE considerations extending the classical Picard iteration to construct a solution. For such solution, we provide a global convergence analysis in mean-field law showing that the solution of the associated nonlinear nonlocal Fokker-Planck equation converges exponentially fast to the unique solution of the bi-level optimization problem provided suitable choices of the hyperparameters. The practicability and efficiency of our CB$^2$O algorithm is demonstrated through extensive numerical experiments in the settings of constrained global optimization, sparse representation learning, and robust (clustered) federated learning.

A discrete Consensus-Based Global Optimization Method with Noisy Objective Function

Consensus-based optimisation with truncated noise

Consensus-Based Optimization Methods Converge Globally

A PDE Framework of Consensus-Based Optimization for Objectives with Multiple Global Minimizers

Consensus under Bounded Noise in Discrete Network Systems: an Algorithm with Fast Convergence and High Accuracy.

Constrained Consensus-Based Optimization and Numerical Heuristics for the Few Particle Regime

Discrete Consensus-Based Optimization

Consensus-Based Optimization for Saddle Point Problems

Leveraging Memory Effects and Gradient Information in Consensus-Based Optimization: On Global Convergence in Mean-Field Law

Convergence analysis of the discrete consensus-based optimization algorithm with random batch interactions and heterogeneous noises

Convergence of Anisotropic Consensus-Based Optimization in Mean-Field Law

Consensus-based algorithms for stochastic optimization problems

Constrained Optimization in the Presence of Noise

Fast and robust consensus-based optimization via optimal feedback control

Gradient is All You Need?

From particle swarm optimization to consensus based optimization: Stochastic modeling and mean-field limit

CB$^2$O: Consensus-Based Bi-Level Optimization

Convergence and error estimates for time-discrete consensus-based optimization algorithms

A Stochastic Semismooth Newton Method for Nonsmooth Nonconvex Optimization.

On the Convergence of Interior-Point Methods for Bound-Constrained Nonlinear Optimization Problems with Noise