Abstract:This paper investigates a class of stochastic bilevel optimization problems where the upper-level function is nonconvex with potentially unbounded smoothness and the lower-level problem is strongly convex. These problems have significant applications in sequential data learning, such as text classification using recurrent neural networks. The unbounded smoothness is characterized by the smoothness constant of the upper-level function scaling linearly with the gradient norm, lacking a uniform upper bound. Existing state-of-the-art algorithms require $\widetilde{O}(1/\epsilon^4)$ oracle calls of stochastic gradient or Hessian/Jacobian-vector product to find an $\epsilon$-stationary point. However, it remains unclear if we can further improve the convergence rate when the assumptions for the function in the population level also hold for each random realization almost surely (e.g., Lipschitzness of each realization of the stochastic gradient). To address this issue, we propose a new Accelerated Bilevel Optimization algorithm named AccBO. The algorithm updates the upper-level variable by normalized stochastic gradient descent with recursive momentum and the lower-level variable by the stochastic Nesterov accelerated gradient descent algorithm with averaging. We prove that our algorithm achieves an oracle complexity of $\widetilde{O}(1/\epsilon^3)$ to find an $\epsilon$-stationary point. Our proof relies on a novel lemma characterizing the dynamics of stochastic Nesterov accelerated gradient descent algorithm under distribution drift with high probability for the lower-level variable, which is of independent interest and also plays a crucial role in analyzing the hypergradient estimation error over time. Experimental results on various tasks confirm that our proposed algorithm achieves the predicted theoretical acceleration and significantly outperforms baselines in bilevel optimization.

Advancing the lower bounds: An accelerated, stochastic, second-order method with optimal adaptation to inexactness

Nesterov's Acceleration For Approximate Newton.

Nonasymptotic Analysis of Accelerated Methods With Inexact Oracle Under Absolute Error Bound

Stochastic Optimization for Non-convex Problem with Inexact Hessian Matrix, Gradient, and Function

Universality of AdaGrad Stepsizes for Stochastic Optimization: Inexact Oracle, Acceleration and Variance Reduction

Faster Convergence of Stochastic Accelerated Gradient Descent under Interpolation

An Accelerated Algorithm for Stochastic Bilevel Optimization under Unbounded Smoothness

Stochastic Optimization for Nonconvex Problem with Inexact Hessian Matrix, Gradient, and Function

Stochastic Newton Proximal Extragradient Method

The "Black-Box" Optimization Problem: Zero-Order Accelerated Stochastic Method via Kernel Approximation

The Black-Box Optimization Problem: Zero-Order Accelerated Stochastic Method via Kernel Approximation

Stochastic Second-order Methods for Non-convex Optimization with Inexact Hessian and Gradient

Exploring Jacobian Inexactness in Second-Order Methods for Variational Inequalities: Lower Bounds, Optimal Algorithms and Quasi-Newton Approximations

Adaptation to Inexactness for some Gradient-type Methods

Adaptive first-order methods with enhanced worst-case rates

Accelerated stochastic approximation with state-dependent noise

Accelerated Almost-Sure Convergence Rates for Nonconvex Stochastic Gradient Descent using Stochastic Learning Rates

Optimal inexactness schedules for tunable oracle-based methods

Intermediate Gradient Methods with Relative Inexactness

Accelerated Convex Optimization with Stochastic Gradients: Generalizing the Strong-Growth Condition

Optimal inexactness schedules for Tunable Oracle based Methods