Abstract:Bilevel optimization is a popular two-level hierarchical optimization, which has been widely applied to many machine learning tasks such as hyperparameter learning, meta learning and continual learning. Although many bilevel optimization methods recently have been developed, the bilevel methods are not well studied when the lower-level problem is nonconvex. To fill this gap, in the paper, we study a class of nonconvex bilevel optimization problems, where both upper-level and lower-level problems are nonconvex, and the lower-level problem satisfies Polyak-Łojasiewicz (PL) condition. We propose an efficient momentum-based gradient bilevel method (MGBiO) to solve these deterministic problems. Meanwhile, we propose a class of efficient momentum-based stochastic gradient bilevel methods (MSGBiO and VR-MSGBiO) to solve these stochastic problems. Moreover, we provide a useful convergence analysis framework for our methods. Specifically, under some mild conditions, we prove that our MGBiO method has a sample (or gradient) complexity of $O(\epsilon^{-2})$ for finding an $\epsilon$-stationary solution of the deterministic bilevel problems (i.e., $\|\nabla F(x)\|\leq \epsilon$), which improves the existing best results by a factor of $O(\epsilon^{-1})$. Meanwhile, we prove that our MSGBiO and VR-MSGBiO methods have sample complexities of $\tilde{O}(\epsilon^{-4})$ and $\tilde{O}(\epsilon^{-3})$, respectively, in finding an $\epsilon$-stationary solution of the stochastic bilevel problems (i.e., $\mathbb{E}\|\nabla F(x)\|\leq \epsilon$), which improves the existing best results by a factor of $\tilde{O}(\epsilon^{-3})$. Extensive experimental results on bilevel PL game and hyper-representation learning demonstrate the efficiency of our algorithms. This paper commemorates the mathematician Boris Polyak (1935 -2023).

On the convergence of proximal gradient methods for convex simple bilevel optimization

Adaptive proximal algorithms for convex optimization under local Lipschitz continuity of the gradient

Linearized Proximal Algorithms with Adaptive Stepsizes for Convex Composite Optimization with Applications

Proximal Gradient-Type Algorithms for a Class of Bilevel Programming Problems

A Conditional Gradient-based Method for Simple Bilevel Optimization with Convex Lower-level Problem

On Convergence Rates of Linearized Proximal Algorithms for Convex Composite Optimization with Applications.

Near-Optimal Convex Simple Bilevel Optimization with a Bisection Method

An Accelerated Gradient Method for Convex Smooth Simple Bilevel Optimization

Proximal Subgradient Algorithm for a Class of Nonconvex Bilevel Equilibrium Problems

A Near-Optimal Algorithm for Convex Simple Bilevel Optimization under Weak Assumptions

Convergence of Nonmonotone Proximal Gradient Methods under the Kurdyka-Lojasiewicz Property without a Global Lipschitz Assumption

On the convergence of adaptive first order methods: proximal gradient and alternating minimization algorithms

Projection-Free Methods for Stochastic Simple Bilevel Optimization with Convex Lower-level Problem

Safeguarding adaptive methods: global convergence of Barzilai-Borwein and other stepsize choices

Barzilai-Borwein Proximal Gradient Methods for Multiobjective Composite Optimization Problems with Improved Linear Convergence

Adaptive proximal gradient methods are universal without approximation

Accelerated Bregman Proximal Gradient Methods for Relatively Smooth Convex Optimization

Convergence of inertial prox-penalization and inertial forward–backward algorithms for solving monotone bilevel equilibrium problems

A Variational Approach on Level sets and Linear Convergence of Variable Bregman Proximal Gradient Method for Nonconvex Optimization Problems

On Momentum-Based Gradient Methods for Bilevel Optimization with Nonconvex Lower-Level

Alternating and Parallel Proximal Gradient Methods for Nonsmooth, Nonconvex Minimax: A Unified Convergence Analysis