Abstract:Bilevel optimization provides a comprehensive framework that bridges single- and multi-objective optimization, encompassing various formulations, including standard nonlinear programs. This paper focuses on a specific class of bilevel optimization known as simple bilevel optimization. In these problems, the objective is to minimize a composite convex function over the optimal solution set of another composite convex minimization problem. By reformulating the simple bilevel problem as finding the left-most root of a nonlinear equation, we employ a bisection scheme to efficiently obtain a solution that is $\epsilon$-optimal for both the upper- and lower-level objectives. In each iteration, the bisection narrows down an interval by assessing the feasibility of a discriminating criterion. By introducing a novel dual approach and employing the Accelerated Proximal Gradient (APG) method, we demonstrate that each subproblem in the bisection scheme can be solved in ${\mathcal{O}}(\sqrt{(L_{g_1}+2D_z L_{f_1}+1)/\epsilon}|\log\epsilon|^2)$ oracle queries under weak assumptions. Here, $L_{f_1}$ and $L_{g_1}$ represent the Lipschitz constants of the gradients of the upper- and lower-level objectives' smooth components, and $D_z$ is the upper bound of the optimal multiplier of the subproblem. Considering the number of binary searches, the total complexity of our proposed method is ${\mathcal{O}}(\sqrt{(L_{g_1}+2D_z L_{f_1}+1)/\epsilon}|\log\epsilon|^3)$. Our method achieves near-optimal complexity results, comparable to those in unconstrained smooth or composite convex optimization when disregarding the logarithmic terms. Numerical experiments also demonstrate the superior performance of our method compared to the state-of-the-art.

A Fast Algorithm for Convex Composite Bi-Level Optimization

Convex Bi-Level Optimization Problems with Non-smooth Outer Objective Function

Linearized Proximal Algorithms with Adaptive Stepsizes for Convex Composite Optimization with Applications

On Convergence Rates of Linearized Proximal Algorithms for Convex Composite Optimization with Applications.

Barzilai-Borwein Proximal Gradient Methods for Multiobjective Composite Optimization Problems with Improved Linear Convergence

Near-Optimal Convex Simple Bilevel Optimization with a Bisection Method

An Accelerated Gradient Method for Convex Smooth Simple Bilevel Optimization

A Near-Optimal Algorithm for Convex Simple Bilevel Optimization under Weak Assumptions

Adaptive Proximal Average Approximation for Composite Convex Minimization.

A FISTA-type accelerated gradient algorithm for solving smooth nonconvex composite optimization problems

A Projection-Free Method for Solving Convex Bilevel Optimization Problems

A Conditional Gradient-based Method for Simple Bilevel Optimization with Convex Lower-level Problem

Accelerated Bregman Proximal Gradient Methods for Relatively Smooth Convex Optimization

Nonconvex Stochastic Bregman Proximal Gradient Method for Nonconvex Composite Problems

Linearization Algorithms for Fully Composite Optimization

Modified Double Inertial Extragradient-like Approaches for Convex Bilevel Optimization Problems with VIP and CFPP Constraints

Linearly-Convergent FISTA Variant for Composite Optimization with Duality

A Proximal Gradient Algorithm for Decentralized Composite Optimization

An Inexact Conditional Gradient Method for Constrained Bilevel Optimization

Optimal Hessian/Jacobian-Free Nonconvex-PL Bilevel Optimization

Constrained Bi-Level Optimization: Proximal Lagrangian Value function Approach and Hessian-free Algorithm