Abstract:Bilevel optimization is an important formulation for many machine learning problems. Current bilevel optimization algorithms assume that the gradient of the upper-level function is Lipschitz. However, recent studies reveal that certain neural networks such as recurrent neural networks (RNNs) and long-short-term memory networks (LSTMs) exhibit potential unbounded smoothness, rendering conventional bilevel optimization algorithms unsuitable. In this paper, we design a new bilevel optimization algorithm, namely BO-REP, to address this challenge. This algorithm updates the upper-level variable using normalized momentum and incorporates two novel techniques for updating the lower-level variable: \textit{initialization refinement} and \textit{periodic updates}. Specifically, once the upper-level variable is initialized, a subroutine is invoked to obtain a refined estimate of the corresponding optimal lower-level variable, and the lower-level variable is updated only after every specific period instead of each iteration. When the upper-level problem is nonconvex and unbounded smooth, and the lower-level problem is strongly convex, we prove that our algorithm requires $\widetilde{\mathcal{O}}(1/\epsilon^4)$ iterations to find an $\epsilon$-stationary point in the stochastic setting, where each iteration involves calling a stochastic gradient or Hessian-vector product oracle. Notably, this result matches the state-of-the-art complexity results under the bounded smoothness setting and without mean-squared smoothness of the stochastic gradient, up to logarithmic factors. Our proof relies on novel technical lemmas for the periodically updated lower-level variable, which are of independent interest. Our experiments on hyper-representation learning, hyperparameter optimization, and data hyper-cleaning for text classification tasks demonstrate the effectiveness of our proposed algorithm.

An Alternating Linearization Method with Inexact Data for Bilevel Nonsmooth Convex Optimization

Near-Optimal Convex Simple Bilevel Optimization with a Bisection Method

A Near-Optimal Algorithm for Convex Simple Bilevel Optimization under Weak Assumptions

An Accelerated Gradient Method for Convex Smooth Simple Bilevel Optimization

Functionally Constrained Algorithm Solves Convex Simple Bilevel Problems

A fresh look at nonsmooth Levenberg–Marquardt methods with applications to bilevel optimization

Penalty-based Methods for Simple Bilevel Optimization under Hölderian Error Bounds

An Inexact Conditional Gradient Method for Constrained Bilevel Optimization

A Cutting Plane and Level Stabilization Bundle Method with Inexact Data for Minimizing Nonsmooth Nonconvex Functions

Semismooth Newton-type method for bilevel optimization: Global convergence and extensive numerical experiments

Inexact bilevel stochastic gradient methods for constrained and unconstrained lower-level problems

Convergence of alternating direction method for minimizing sum of two nonconvex functions with linear constraints

Linear Convergence of Inexact Descent Method and Inexact Proximal Gradient Algorithms for Lower-Order Regularization Problems

Theoretical smoothing frameworks for general nonsmooth bilevel problems

A Generalized Alternating Method for Bilevel Learning under the Polyak-Łojasiewicz Condition

Modified Double Inertial Extragradient-like Approaches for Convex Bilevel Optimization Problems with VIP and CFPP Constraints

Smoothing SQP methods for solving degenerate nonsmooth constrained optimization problems with applications to bilevel programs

A Projection-Free Method for Solving Convex Bilevel Optimization Problems

Bilevel Optimization under Unbounded Smoothness: A New Algorithm and Convergence Analysis

Enhanced Barrier-Smoothing Technique for Bilevel Optimization with Nonsmooth Mappings

Complexity of linearized quadratic penalty for optimization with nonlinear equality constraints