Abstract:This paper investigates a class of stochastic bilevel optimization problems where the upper-level function is nonconvex with potentially unbounded smoothness and the lower-level problem is strongly convex. These problems have significant applications in sequential data learning, such as text classification using recurrent neural networks. The unbounded smoothness is characterized by the smoothness constant of the upper-level function scaling linearly with the gradient norm, lacking a uniform upper bound. Existing state-of-the-art algorithms require $\widetilde{O}(1/\epsilon^4)$ oracle calls of stochastic gradient or Hessian/Jacobian-vector product to find an $\epsilon$-stationary point. However, it remains unclear if we can further improve the convergence rate when the assumptions for the function in the population level also hold for each random realization almost surely (e.g., Lipschitzness of each realization of the stochastic gradient). To address this issue, we propose a new Accelerated Bilevel Optimization algorithm named AccBO. The algorithm updates the upper-level variable by normalized stochastic gradient descent with recursive momentum and the lower-level variable by the stochastic Nesterov accelerated gradient descent algorithm with averaging. We prove that our algorithm achieves an oracle complexity of $\widetilde{O}(1/\epsilon^3)$ to find an $\epsilon$-stationary point. Our proof relies on a novel lemma characterizing the dynamics of stochastic Nesterov accelerated gradient descent algorithm under distribution drift with high probability for the lower-level variable, which is of independent interest and also plays a crucial role in analyzing the hypergradient estimation error over time. Experimental results on various tasks confirm that our proposed algorithm achieves the predicted theoretical acceleration and significantly outperforms baselines in bilevel optimization.

A Note on Nesterov's Accelerated Method in Nonconvex Optimization: a Weak Estimate Sequence Approach

A Lyapunov analysis of accelerated methods in optimization

Accelerating Nesterov's Method for Strongly Convex Functions with Lipschitz Gradient

On adapting Nesterov's scheme to accelerate iterative methods for linear problems

Nesterov's Acceleration For Approximate Newton.

The Global R-linear Convergence of Nesterov's Accelerated Gradient Method with Unknown Strongly Convex Parameter

Stagewise Accelerated Stochastic Gradient Methods for Nonconvex Optimization

A Nesterov's Accelerated Projected Gradient Method for Monotone Variational Inequalities

An Accelerated Distributed Method with Inexact Model of Relative Smoothness and Strong Convexity

Accelerated gradient methods for nonconvex optimization: Escape trajectories from strict saddle points and convergence to local minima

A Stochastic Nesterov’s Smoothing Accelerated Method for General Nonsmooth Constrained Stochastic Composite Convex Optimization

The Nesterov-Spokoiny Acceleration: $o(1/k^2)$ Convergence without Proximal Operations

Faster Convergence of Stochastic Accelerated Gradient Descent under Interpolation

An Accelerated Algorithm for Stochastic Bilevel Optimization under Unbounded Smoothness

Accelerated Quasi-Newton Proximal Extragradient: Faster Rate for Smooth Convex Optimization

Fast Multiobjective Gradient Methods with Nesterov Acceleration via Inertial Gradient-Like Systems

Tradeoffs between convergence rate and noise amplification for momentum-based accelerated optimization algorithms

Nesterov acceleration in benignly non-convex landscapes

A Unified Scheme to Accelerate Adaptive Cubic Regularization and Gradient Methods for Convex Optimization

Lyapunov Analysis For Monotonically Forward-Backward Accelerated Algorithms

Faster Acceleration for Steepest Descent