Abstract:As we know, the stepsize is extremely crucial to gradient method. A new type of stepsize is introduced for the gradient method in the paper, which is generated by minimizing the norm of the approximate model of the gradient along the line of negative gradient direction. Based on the retard technique, we present a new gradient method by minimizing adaptively the approximate model of the objective function and the norm of the approximate model of the gradient along the line of negative gradient direction for strictly convex quadratic mini-mization. The convergence of the proposed method is estab-lished. The numerical experiments on four groups of strictly convex quadratic minimization problems illustrate that the proposed method is very promising. We also extend the new gradient method for convex quadratic minimization to general unconstrained optimization by incorporating a nonmonotone line search. The convergence of the resulting method is estab-lished. The numerical experiment on the 147 test functions from the CUTEst library indicates that the resulting method is superior to some efficient gradient methods including the BBQ method (SIAM J Optim. 2021;31(4):3068-3096) and is com-petitive to two famous conjugate gradient software packages CGOPT (1.0) and CG_DESCENT (5.0).

Gradient descent with a general cost

Generalization to the Natural Gradient Descent

Mirror and Preconditioned Gradient Descent in Wasserstein Space

A Riemannian Gradient Sampling Algorithm for Nonsmooth Optimization on Manifolds

Convergence and Trade-Offs in Riemannian Gradient Descent and Riemannian Proximal Point

Gauss-Southwell type descent methods for low-rank matrix optimization

Convergence Analysis of Gradient Algorithms on Riemannian Manifolds Without Curvature Constraints and Application to Riemannian Mass

Convergence of Gradient Algorithms for Nonconvex C1+α Cost Functions

Laplacian Smoothing Gradient Descent

Super Gradient Descent: Global Optimization requires Global Gradient

New Gradient Methods with Adaptive Stepsizes by Approximate Models

Optimizing $(L_0, L_1)$-Smooth Functions by Gradient Methods

Accelerated Objective Gap and Gradient Norm Convergence for Gradient Descent via Long Steps

Tropical Gradient Descent

Provably Faster Gradient Descent via Long Steps

Optimal sampling for stochastic and natural gradient descent

Parameter-Free Accelerated Gradient Descent for Nonconvex Minimization

Toward a Unified Theory of Gradient Descent under Generalized Smoothness

Inexact Riemannian Gradient Descent Method for Nonconvex Optimization

Gradient-type penalty method with inertial effects for solving constrained convex optimization problems with smooth data

A gradient descent akin method for constrained optimization: algorithms and applications