Abstract:We propose novel optimal and parameter-free algorithms for computing an approximate solution with small (projected) gradient norm. Specifically, for computing an approximate solution such that the norm of its (projected) gradient does not exceed $\varepsilon$, we obtain the following results: a) for the convex case, the total number of gradient evaluations is bounded by $O(1)\sqrt{L\|x_0 - x^*\|/\varepsilon}$, where $L$ is the Lipschitz constant of the gradient and $x^*$ is any optimal solution; b) for the strongly convex case, the total number of gradient evaluations is bounded by $O(1)\sqrt{L/\mu}\log(\|\nabla f(x_0)\|/\epsilon)$, where $\mu$ is the strong convexity modulus; and c) for the nonconvex case, the total number of gradient evaluations is bounded by $O(1)\sqrt{Ll}(f(x_0) - f(x^*))/\varepsilon^2$, where $l$ is the lower curvature constant. Our complexity results match the lower complexity bounds of the convex and strongly cases, and achieve the above best-known complexity bound for the nonconvex case for the first time in the literature. Moreover, for all the convex, strongly convex, and nonconvex cases, we propose parameter-free algorithms that do not require the input of any problem parameters. To the best of our knowledge, there do not exist such parameter-free methods before especially for the strongly convex and nonconvex cases. Since most regularity conditions (e.g., strong convexity and lower curvature) are imposed over a global scope, the corresponding problem parameters are notoriously difficult to estimate. However, gradient norm minimization equips us with a convenient tool to monitor the progress of algorithms and thus the ability to estimate such parameters in-situ.

New Gradient Methods with Adaptive Stepsizes by Approximate Models

Two efficient gradient methods with approximately optimal stepsizes based on regularization models for unconstrained optimization

An Improved Gradient Method with Approximately Optimal Stepsize Based on Conic model for Unconstrained Optimization

Gradient Methods with Adaptive Step-Sizes

An Adaptive Stochastic Gradient Method with Non-negative Gauss-Newton Stepsizes

A nonmonotone accelerated proximal gradient method with variable stepsize strategy for nonsmooth and nonconvex minimization problems

AGDA+: Proximal Alternating Gradient Descent Ascent Method With a Nonmonotone Adaptive Step-Size Search For Nonconvex Minimax Problems

Provably Faster Gradient Descent via Long Steps

Accelerated Objective Gap and Gradient Norm Convergence for Gradient Descent via Long Steps

A class of modified accelerated proximal gradient methods for nonsmooth and nonconvex minimization problems

Directional Smoothness and Gradient Methods: Convergence and Adaptivity

On the acceleration of the Barzilai-Borwein method

Accelerated Gradient Descent via Long Steps

A New Dai-Liao Conjugate Gradient Method based on Approximately Optimal Stepsize for Unconstrained Optimization

A stochastic variance reduced gradient method with adaptive step for stochastic optimization

Parameter-Free Accelerated Gradient Descent for Nonconvex Minimization

Universality of AdaGrad Stepsizes for Stochastic Optimization: Inexact Oracle, Acceleration and Variance Reduction

Safeguarding adaptive methods: global convergence of Barzilai-Borwein and other stepsize choices

Optimal and parameter-free gradient minimization methods for convex and nonconvex optimization

Adaptive Step Sizes for Preconditioned Stochastic Gradient Descent