Abstract:Bilevel optimization (BO) has recently gained prominence in many machine learning applications due to its ability to capture the nested structure inherent in these problems. Recently, many hypergradient methods have been proposed as effective solutions for solving large-scale problems. However, current hypergradient methods for the lower-level constrained bilevel optimization (LCBO) problems need very restrictive assumptions, namely, where optimality conditions satisfy the differentiability and invertibility conditions and lack a solid analysis of the convergence rate. What's worse, existing methods require either double-loop updates, which are sometimes less efficient. To solve this problem, in this paper, we propose a new hypergradient of LCBO leveraging the theory of nonsmooth implicit function theorem instead of using the restrive assumptions. In addition, we propose a \textit{single-loop single-timescale} algorithm based on the double-momentum method and adaptive step size method and prove it can return a $(\delta, \epsilon)$-stationary point with $\tilde{\mathcal{O}}(d_2^2\epsilon^{-4})$ iterations. Experiments on two applications demonstrate the effectiveness of our proposed method.

A Nonmonotone Broyden Method for Unconstrained Optimization

The convergence of conjugate gradient method with nonmonotone line search

Global Convergence Analysis of a New Nonmonotone BFGS Algorithm on Convex Objective Functions

An Efficient Interior-Point Algorithm with New Non-Monotone Line Search Filter Method for Nonlinear Constrained Programming

An improved Dai‐Liao‐style hybrid conjugate gradient‐based method for solving unconstrained nonconvex optimization and extension to constrained nonlinear monotone equations

Convergence of the Nonmonotone Perry and Shanno Method for Optimization

A Branch-and-Bound Based Method for Solving Monotone Optimization Problems

A hybrid BFGS-Like method for monotone operator equations with applications

A New Version Coefficient Of Three-Term Conjugate Gradient Method To Solve Unconstrained Optimization

A new nonmonotone trust-region method of conic model for solving unconstrained optimization

A two-step Broyden-like method for nonlinear equations

A nonmonotone trust region method for unconstrained optimization

A Nonmonotone Filter Barzilai-Borwein Method for Optimization

Global Convergence of the Polak-Ribière and Hestenes-Stiefel Conjugate Gradient Methods for the Unconstrained Nonlinear Optimization

A Conjugate Gradient Method with Global Convergence for Large-Scale Unconstrained Optimization Problems

Barzilai and Borwein conjugate gradient method equipped with a non-monotone line search technique and its application on non-negative matrix factorization

Nonmonotone Gradient Methods for Vector Optimization with a Portfolio Optimization Application

A class of nonmonotone Armijo-type line search method for unconstrained optimization

A Mixed and Superlinearly Convergent Algorithm for Constrained Optimization.

Double Momentum Method for Lower-Level Constrained Bilevel Optimization

Block Broyden's Methods for Solving Nonlinear Equations.