Abstract:This paper investigates a class of stochastic bilevel optimization problems where the upper-level function is nonconvex with potentially unbounded smoothness and the lower-level problem is strongly convex. These problems have significant applications in sequential data learning, such as text classification using recurrent neural networks. The unbounded smoothness is characterized by the smoothness constant of the upper-level function scaling linearly with the gradient norm, lacking a uniform upper bound. Existing state-of-the-art algorithms require $\widetilde{O}(1/\epsilon^4)$ oracle calls of stochastic gradient or Hessian/Jacobian-vector product to find an $\epsilon$-stationary point. However, it remains unclear if we can further improve the convergence rate when the assumptions for the function in the population level also hold for each random realization almost surely (e.g., Lipschitzness of each realization of the stochastic gradient). To address this issue, we propose a new Accelerated Bilevel Optimization algorithm named AccBO. The algorithm updates the upper-level variable by normalized stochastic gradient descent with recursive momentum and the lower-level variable by the stochastic Nesterov accelerated gradient descent algorithm with averaging. We prove that our algorithm achieves an oracle complexity of $\widetilde{O}(1/\epsilon^3)$ to find an $\epsilon$-stationary point. Our proof relies on a novel lemma characterizing the dynamics of stochastic Nesterov accelerated gradient descent algorithm under distribution drift with high probability for the lower-level variable, which is of independent interest and also plays a crucial role in analyzing the hypergradient estimation error over time. Experimental results on various tasks confirm that our proposed algorithm achieves the predicted theoretical acceleration and significantly outperforms baselines in bilevel optimization.

Theoretical smoothing frameworks for general nonsmooth bilevel problems

Enhanced Barrier-Smoothing Technique for Bilevel Optimization with Nonsmooth Mappings

Bilevel Optimization under Unbounded Smoothness: A New Algorithm and Convergence Analysis

A framework for bilevel optimization that enables stochastic and global variance reduction algorithms

A fresh look at nonsmooth Levenberg–Marquardt methods with applications to bilevel optimization

Semismooth Newton-type method for bilevel optimization: Global convergence and extensive numerical experiments

Smoothing SQP methods for solving degenerate nonsmooth constrained optimization problems with applications to bilevel programs

Barrier Function for Bilevel Optimization with Coupled Lower-Level Constraints: Formulation, Approximation and Algorithms

Fully Zeroth-Order Bilevel Programming via Gaussian Smoothing

Inexact bilevel stochastic gradient methods for constrained and unconstrained lower-level problems

An Accelerated Algorithm for Stochastic Bilevel Optimization under Unbounded Smoothness

An Inexact Primal-Dual Smoothing Framework for Large-Scale Non-Bilinear Saddle Point Problems

Moreau Envelope Based Difference-of-weakly-Convex Reformulation and Algorithm for Bilevel Programs

Penalty-based Methods for Simple Bilevel Optimization under Hölderian Error Bounds

Optimal Algorithms for Stochastic Bilevel Optimization under Relaxed Smoothness Conditions

Functionally Constrained Algorithm Solves Convex Simple Bilevel Problems

Double Momentum Method for Lower-Level Constrained Bilevel Optimization

Regular Subgradients of Marginal Functions with Applications to Calculus and Bilevel Programming

A new problem qualification based on approximate KKT conditions for Lipschitzian optimization with application to bilevel programming

An Accelerated Gradient Method for Convex Smooth Simple Bilevel Optimization

Single-loop Projection-free and Projected Gradient-based Algorithms for Nonconvex-concave Saddle Point Problems with Bilevel Structure