Abstract:A significant milestone in modern gradient-based optimization was achieved with the development of Nesterov's accelerated gradient descent (NAG) method. This forward-backward technique has been further advanced with the introduction of its proximal generalization, commonly known as the fast iterative shrinkage-thresholding algorithm (FISTA), which enjoys widespread application in image science and engineering. Nonetheless, it remains unclear whether both NAG and FISTA exhibit linear convergence for strongly convex functions. Remarkably, these algorithms demonstrate convergence without requiring any prior knowledge of strongly convex modulus, and this intriguing characteristic has been acknowledged as an open problem in the comprehensive review [Chambolle and Pock, 2016, Appendix B]. In this paper, we address this question by utilizing the high-resolution ordinary differential equation (ODE) framework. Expanding upon the established phase-space representation, we emphasize the distinctive approach employed in crafting the Lyapunov function, which involves a dynamically adapting coefficient of kinetic energy that evolves throughout the iterations. Furthermore, we highlight that the linear convergence of both NAG and FISTA is independent of the parameter $r$. Additionally, we demonstrate that the square of the proximal subgradient norm likewise advances towards linear convergence.

A look at the convergence of the augmented Lagrange method for nondifferentiable convex programming from the view of a gradient method with constant stepsize

The convergence of conjugate gradient method with nonmonotone line search

A Unified Convergence Analysis of a Second-Order Method of Multipliers for Nonlinear Conic Programming

On the Convergence of Augmented Lagrangian Methods for Constrained Global Optimization

Convergence analysis of the Gauss–Newton method for convex inclusion and convex-composite optimization problems

Convergence Analysis of Gradient Algorithms on Riemannian Manifolds Without Curvature Constraints and Application to Riemannian Mass

Accelerated gradient methods for nonconvex nonlinear and stochastic programming

A New Insight on Augmented Lagrangian Method with Applications in Machine Learning

A gradient descent akin method for constrained optimization: algorithms and applications

Augmented Lagrangian method for nonlinear circular conic programs: a local convergence analysis

Convergence of alternating direction method for minimizing sum of two nonconvex functions with linear constraints

Developing Lagrangian-based Methods for Nonsmooth Nonconvex Optimization

A Primal Majorized Semismooth Newton-CG Augmented Lagrangian Method for Large-Scale Linearly Constrained Convex Programming

Inexact Augmented Lagrangian Methods for Conic Programs: Quadratic Growth and Linear Convergence

An improved Dai‐Liao‐style hybrid conjugate gradient‐based method for solving unconstrained nonconvex optimization and extension to constrained nonlinear monotone equations

Convergence of a Recurrent Neural Network for Nonconvex Optimization Based on an Augmented Lagrangian Function

Convergence of the Descent Nonlinear Conjugate Gradient Methods

Accelerated Proximal Gradient Methods for Nonconvex Programming

Global Convergence of Langevin Dynamics Based Algorithms for Nonconvex Optimization

Linear convergence of forward-backward accelerated algorithms without knowledge of the modulus of strong convexity

A New Inexact Gradient Descent Method with Applications to Nonsmooth Convex Optimization