Abstract:This paper investigates a class of stochastic bilevel optimization problems where the upper-level function is nonconvex with potentially unbounded smoothness and the lower-level problem is strongly convex. These problems have significant applications in sequential data learning, such as text classification using recurrent neural networks. The unbounded smoothness is characterized by the smoothness constant of the upper-level function scaling linearly with the gradient norm, lacking a uniform upper bound. Existing state-of-the-art algorithms require $\widetilde{O}(1/\epsilon^4)$ oracle calls of stochastic gradient or Hessian/Jacobian-vector product to find an $\epsilon$-stationary point. However, it remains unclear if we can further improve the convergence rate when the assumptions for the function in the population level also hold for each random realization almost surely (e.g., Lipschitzness of each realization of the stochastic gradient). To address this issue, we propose a new Accelerated Bilevel Optimization algorithm named AccBO. The algorithm updates the upper-level variable by normalized stochastic gradient descent with recursive momentum and the lower-level variable by the stochastic Nesterov accelerated gradient descent algorithm with averaging. We prove that our algorithm achieves an oracle complexity of $\widetilde{O}(1/\epsilon^3)$ to find an $\epsilon$-stationary point. Our proof relies on a novel lemma characterizing the dynamics of stochastic Nesterov accelerated gradient descent algorithm under distribution drift with high probability for the lower-level variable, which is of independent interest and also plays a crucial role in analyzing the hypergradient estimation error over time. Experimental results on various tasks confirm that our proposed algorithm achieves the predicted theoretical acceleration and significantly outperforms baselines in bilevel optimization.

Unbiased Quasi-Hyperbolic Nesterov-Gradient Momentum-Based Optimizers for Accelerating Convergence

Accelerated Optimization in Deep Learning with a Proportional-Integral-derivative Controller

Stochastic Momentum Method with Double Acceleration for Regularized Empirical Risk Minimization

Training Deep Neural Networks with Adaptive Momentum Inspired by the Quadratic Optimization

Multi-stage stochastic gradient method with momentum acceleration

Accelerated Gradient-free Neural Network Training by Multi-convex Alternating Optimization

Stochastic Gradient Descent with Nonlinear Conjugate Gradient-Style Adaptive Momentum

Stagewise Accelerated Stochastic Gradient Methods for Nonconvex Optimization

Unified Convergence Analysis of Stochastic Momentum Methods for Convex and Non-convex Optimization

Adaptive momentum with discriminative weight for neural network stochastic optimization

Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models

Optimal Adaptive and Accelerated Stochastic Gradient Descent

Combining Conjugate Gradient and Momentum for Unconstrained Stochastic Optimization With Applications to Machine Learning

Provable Acceleration of Nesterov's Accelerated Gradient Method over Heavy Ball Method in Training Over-Parameterized Neural Networks

Double Momentum Method for Lower-Level Constrained Bilevel Optimization

A comparative study of recently deep learning optimizers

Gradient Descent Optimization in Deep Learning Model Training Based on Multistage and Method Combination Strategy

Bidirectional Looking with A Novel Double Exponential Moving Average to Adaptive and Non-adaptive Momentum Optimizers

Momentum is All You Need for Data-Driven Adaptive Optimization

State Space Representation and Phase Analysis of Gradient Descent Optimizers.

An Accelerated Algorithm for Stochastic Bilevel Optimization under Unbounded Smoothness