Abstract:This paper investigates a class of stochastic bilevel optimization problems where the upper-level function is nonconvex with potentially unbounded smoothness and the lower-level problem is strongly convex. These problems have significant applications in sequential data learning, such as text classification using recurrent neural networks. The unbounded smoothness is characterized by the smoothness constant of the upper-level function scaling linearly with the gradient norm, lacking a uniform upper bound. Existing state-of-the-art algorithms require $\widetilde{O}(1/\epsilon^4)$ oracle calls of stochastic gradient or Hessian/Jacobian-vector product to find an $\epsilon$-stationary point. However, it remains unclear if we can further improve the convergence rate when the assumptions for the function in the population level also hold for each random realization almost surely (e.g., Lipschitzness of each realization of the stochastic gradient). To address this issue, we propose a new Accelerated Bilevel Optimization algorithm named AccBO. The algorithm updates the upper-level variable by normalized stochastic gradient descent with recursive momentum and the lower-level variable by the stochastic Nesterov accelerated gradient descent algorithm with averaging. We prove that our algorithm achieves an oracle complexity of $\widetilde{O}(1/\epsilon^3)$ to find an $\epsilon$-stationary point. Our proof relies on a novel lemma characterizing the dynamics of stochastic Nesterov accelerated gradient descent algorithm under distribution drift with high probability for the lower-level variable, which is of independent interest and also plays a crucial role in analyzing the hypergradient estimation error over time. Experimental results on various tasks confirm that our proposed algorithm achieves the predicted theoretical acceleration and significantly outperforms baselines in bilevel optimization.

What problem does this paper attempt to address?

This paper aims to solve a class of stochastic bilevel optimization problems, where the upper - level function is non - convex and may have unbounded smoothness, while the lower - level problem is strongly convex. This kind of problems has important applications in sequential data learning, such as text classification using recurrent neural networks. Unbounded smoothness means that the smoothness constant of the upper - level function grows linearly with the gradient norm and lacks a uniform upper bound. The existing state - of - the - art algorithms require $\tilde{O}(\epsilon^{-4})$ calls of stochastic gradients or Hessian / Jacobian vector products to find an $\epsilon$-stationary point. However, it is still unclear whether the convergence rate can be further improved when each stochastic realization almost surely satisfies the assumptions of the population - level function (for example, the Lipschitz property of each stochastic gradient realization). To address this problem, the authors propose a new accelerated bilevel optimization algorithm - AccBO. This algorithm updates the upper - level variables through normalized stochastic gradient descent with recursive momentum, and updates the lower - level variables through the stochastic Nesterov - accelerated gradient - descent algorithm with averaging. The authors prove that this algorithm can find an $\epsilon$-stationary point within the complexity of $\tilde{O}(\epsilon^{-3})$. This result depends on a novel lemma, which describes the dynamic behavior of the stochastic Nesterov - accelerated gradient - descent algorithm under distribution drift and plays an important role in analyzing the change of the hyper - gradient estimation error over time. The experimental results show that the proposed algorithm not only achieves the theoretically predicted acceleration effect, but also significantly outperforms the baseline methods in bilevel optimization tasks.

An Accelerated Algorithm for Stochastic Bilevel Optimization under Unbounded Smoothness

A Nearly Optimal Single Loop Algorithm for Stochastic Bilevel Optimization under Unbounded Smoothness

Bilevel Optimization under Unbounded Smoothness: A New Algorithm and Convergence Analysis

Optimal Algorithms for Stochastic Bilevel Optimization under Relaxed Smoothness Conditions

Double Momentum Method for Lower-Level Constrained Bilevel Optimization

Projection-Free Methods for Stochastic Simple Bilevel Optimization with Convex Lower-level Problem

A New Algorithm With Lower Complexity for Bilevel Optimization

Randomized Stochastic Variance-Reduced Methods for Multi-Task Stochastic Bilevel Optimization

On Penalty Methods for Nonconvex Bilevel Optimization and First-Order Stochastic Approximation

A Loopless Distributed Algorithm for Personalized Bilevel Optimization

Distributed Stochastic Bilevel Optimization: Improved Complexity and Heterogeneity Analysis

An Accelerated Gradient Method for Convex Smooth Simple Bilevel Optimization

Blockwise Stochastic Variance-Reduced Methods with Parallel Speedup for Multi-Block Bilevel Optimization

On Momentum-Based Gradient Methods for Bilevel Optimization with Nonconvex Lower-Level

Towards Extremely Fast Bilevel Optimization with Self-governed Convergence Guarantees

A framework for bilevel optimization that enables stochastic and global variance reduction algorithms

Asynchronous Distributed Bilevel Optimization

On the Complexity of First-Order Methods in Stochastic Bilevel Optimization

Stagewise Accelerated Stochastic Gradient Methods for Nonconvex Optimization

Averaged Method of Multipliers for Bi-Level Optimization without Lower-Level Strong Convexity

Fine-grained Analysis of Stability and Generalization for Stochastic Bilevel Optimization