Abstract:This paper investigates a class of stochastic bilevel optimization problems where the upper-level function is nonconvex with potentially unbounded smoothness and the lower-level problem is strongly convex. These problems have significant applications in sequential data learning, such as text classification using recurrent neural networks. The unbounded smoothness is characterized by the smoothness constant of the upper-level function scaling linearly with the gradient norm, lacking a uniform upper bound. Existing state-of-the-art algorithms require $\widetilde{O}(1/\epsilon^4)$ oracle calls of stochastic gradient or Hessian/Jacobian-vector product to find an $\epsilon$-stationary point. However, it remains unclear if we can further improve the convergence rate when the assumptions for the function in the population level also hold for each random realization almost surely (e.g., Lipschitzness of each realization of the stochastic gradient). To address this issue, we propose a new Accelerated Bilevel Optimization algorithm named AccBO. The algorithm updates the upper-level variable by normalized stochastic gradient descent with recursive momentum and the lower-level variable by the stochastic Nesterov accelerated gradient descent algorithm with averaging. We prove that our algorithm achieves an oracle complexity of $\widetilde{O}(1/\epsilon^3)$ to find an $\epsilon$-stationary point. Our proof relies on a novel lemma characterizing the dynamics of stochastic Nesterov accelerated gradient descent algorithm under distribution drift with high probability for the lower-level variable, which is of independent interest and also plays a crucial role in analyzing the hypergradient estimation error over time. Experimental results on various tasks confirm that our proposed algorithm achieves the predicted theoretical acceleration and significantly outperforms baselines in bilevel optimization.

On the Accelerated Extremum Seeking Driven by Optimal Stochastic Perturbations

Time-delay Induced Stochastic Optimization and Extremum Seeking

The Optimality of (Accelerated) SGD for High-Dimensional Quadratic Optimization

Optimization of Convex Functions with Random Pursuit

Gradient-based Stochastic Extremum Seeking for Static Maps with Delays

Stagewise Accelerated Stochastic Gradient Methods for Nonconvex Optimization

Almost Sure Convergence of Randomised‐difference Descent Algorithm for Stochastic Convex Optimisation

Accelerated stochastic approximation with state-dependent noise

On The Unified Design Of Accelerated Gradient Descent

Accelerated Gradient Algorithms with Adaptive Subspace Search for Instance-Faster Optimization

Optimal Adaptive and Accelerated Stochastic Gradient Descent

A Gradient Smoothed Functional Algorithm with Truncated Cauchy Random Perturbations for Stochastic Optimization

Stochastic Methods in Variational Inequalities: Ergodicity, Bias and Refinements

Deterministic and Stochastic Accelerated Gradient Method for Convex Semi-Infinite Optimization

Accelerated Gradient Descent Driven by Lévy Perturbations

An Accelerated Algorithm for Stochastic Bilevel Optimization under Unbounded Smoothness

An Algebraically Converging Stochastic Gradient Descent Algorithm for Global Optimization

On the acceleration of the Barzilai-Borwein method

Simple and Optimal Stochastic Gradient Methods for Nonsmooth Nonconvex Optimization

Beyond Convexity: Stochastic Quasi-Convex Optimization

Two efficient gradient methods with approximately optimal stepsizes based on regularization models for unconstrained optimization