Abstract:Bi-level optimization problems, where one wishes to find the global minimizer of an upper-level objective function over the globally optimal solution set of a lower-level objective, arise in a variety of scenarios throughout science and engineering, machine learning, and artificial intelligence. In this paper, we propose and investigate, analytically and experimentally, consensus-based bi-level optimization (CB$^2$O), a multi-particle metaheuristic derivative-free optimization method designed to solve bi-level optimization problems when both objectives may be nonconvex. Our method leverages within the computation of the consensus point a carefully designed particle selection principle implemented through a suitable choice of a quantile on the level of the lower-level objective, together with a Laplace principle-type approximation w.r.t. the upper-level objective function, to ensure that the bi-level optimization problem is solved in an intrinsic manner. We give an existence proof of solutions to a corresponding mean-field dynamics, for which we first establish the stability of our consensus point w.r.t. a combination of Wasserstein and $L^2$ perturbations, and consecutively resort to PDE considerations extending the classical Picard iteration to construct a solution. For such solution, we provide a global convergence analysis in mean-field law showing that the solution of the associated nonlinear nonlocal Fokker-Planck equation converges exponentially fast to the unique solution of the bi-level optimization problem provided suitable choices of the hyperparameters. The practicability and efficiency of our CB$^2$O algorithm is demonstrated through extensive numerical experiments in the settings of constrained global optimization, sparse representation learning, and robust (clustered) federated learning.

Convergence of a First-Order Consensus-Based Global Optimization Algorithm

Consensus-Based Optimization Methods Converge Globally

Convergence and error estimates for time-discrete consensus-based optimization algorithms

A Consensus-Based Global Optimization Method with Adaptive Momentum Estimation

A PDE Framework of Consensus-Based Optimization for Objectives with Multiple Global Minimizers

Leveraging Memory Effects and Gradient Information in Consensus-Based Optimization: On Global Convergence in Mean-Field Law

A Consensus-Based Global Optimization Method for High Dimensional Machine Learning Problems

Convergence analysis of the discrete consensus-based optimization algorithm with random batch interactions and heterogeneous noises

Convergence of Anisotropic Consensus-Based Optimization in Mean-Field Law

Discrete Consensus-Based Optimization

Fast and robust consensus-based optimization via optimal feedback control

Constrained Consensus-Based Optimization and Numerical Heuristics for the Few Particle Regime

Gradient is All You Need?

Uniform-in-time mean-field limit estimate for the Consensus-Based Optimization

A discrete Consensus-Based Global Optimization Method with Noisy Objective Function

Convergence of Inexact Steepest Descent Algorithm for Multiobjective Optimizations on Riemannian Manifolds Without Curvature Constraints

CB$^2$O: Consensus-Based Bi-Level Optimization

Consensus-based optimisation with truncated noise

Trends in Consensus-Based Optimization

A Bi-level Globalization Strategy for Non-convex Consensus ADMM and ALADIN

Consensus-Based Optimization for Saddle Point Problems