Directional descent

Andrew J. Young

2024-08-26

Abstract:We identity the optimal non-infinitesimal direction of descent for a convex function. An algorithm is developed that can theoretically minimize a subset of (non-convex) Lipschitz functions.

Optimization and Control

What problem does this paper attempt to address?

Directional Smoothness and Gradient Methods: Convergence and Adaptivity

Aaron Mishkin,Ahmed Khaled,Yuanhao Wang,Aaron Defazio,Robert M. Gower

2024-03-07

Abstract:We develop new sub-optimality bounds for gradient descent (GD) that depend on the conditioning of the objective along the path of optimization, rather than on global, worst-case constants. Key to our proofs is directional smoothness, a measure of gradient variation that we use to develop upper-bounds on the objective. Minimizing these upper-bounds requires solving implicit equations to obtain a sequence of strongly adapted step-sizes; we show that these equations are straightforward to solve for convex quadratics and lead to new guarantees for two classical step-sizes. For general functions, we prove that the Polyak step-size and normalized GD obtain fast, path-dependent rates despite using no knowledge of the directional smoothness. Experiments on logistic regression show our convergence guarantees are tighter than the classical theory based on L-smoothness.

Machine Learning,Optimization and Control
Derivation of Coordinate Descent Algorithms from Optimal Control Theory

I. M. Ross

DOI: https://doi.org/10.1007/s43069-023-00215-6

2023-09-08

Abstract:Recently, it was posited that disparate optimization algorithms may be coalesced in terms of a central source emanating from optimal control theory. Here we further this proposition by showing how coordinate descent algorithms may be derived from this emerging new principle. In particular, we show that basic coordinate descent algorithms can be derived using a maximum principle and a collection of max functions as "control" Lyapunov functions. The convergence of the resulting coordinate descent algorithms is thus connected to the controlled dissipation of their corresponding Lyapunov functions. The operational metric for the search vector in all cases is given by the Hessian of the convex objective function.

Optimization and Control,Machine Learning
A Descent Method for Nonsmooth Multiobjective Optimization in Hilbert Spaces

Konstantin Sonntag,Bennet Gebken,Georg Müller,Sebastian Peitz,Stefan Volkwein

DOI: https://doi.org/10.1007/s10957-024-02520-4

2024-09-14

Journal of Optimization Theory and Applications

Abstract:The efficient optimization method for locally Lipschitz continuous multiobjective optimization problems from Gebken and Peitz (J Optim Theory Appl 188:696–723, 2021) is extended from finite-dimensional problems to general Hilbert spaces. The method iteratively computes Pareto critical points, where in each iteration, an approximation of the Clarke subdifferential is computed in an efficient manner and then used to compute a common descent direction for all objective functions. To prove convergence, we present some new optimality results for nonsmooth multiobjective optimization problems in Hilbert spaces. Using these, we can show that every accumulation point of the sequence generated by our algorithm is Pareto critical under common assumptions. Computational efficiency for finding Pareto critical points is numerically demonstrated for multiobjective optimal control of an obstacle problem.

mathematics, applied,operations research & management science
A descent subgradient method using Mifflin line search for nonsmooth nonconvex optimization

Morteza Maleknia,Majid Soleimani-damaneh

2023-04-08

Abstract:We propose a descent subgradient algorithm for minimizing a real function, assumed to be locally Lipschitz, but not necessarily smooth or convex. To find an effective descent direction, the Goldstein subdifferential is approximated through an iterative process. The method enjoys a new two-point variant of Mifflin line search in which the subgradients are arbitrary. Thus, the line search procedure is easy to implement. Moreover, in comparison to bundle methods, the quadratic subproblems have a simple structure, and to handle nonconvexity the proposed method requires no algorithmic modification. We study the global convergence of the method and prove that any accumulation point of the generated sequence is Clarke stationary, assuming that the objective $f$ is weakly upper semismooth. We illustrate the efficiency and effectiveness of the proposed algorithm on a collection of academic and semi-academic test problems.

Optimization and Control
Regularized Asymptotic Descents for a Class of Nonconvex Optimization Problems.

Xiaopeng Luo,Xin Xu

2020-01-01

Abstract:We propose and analyze regularized asymptotic descent (RAD) methods for finding the global minimizer of a class of possibly nonconvex, nonsmooth, or even discontinuous functions. Such functions are extended from strongly convex functions with Lipschitz-continuous gradients. We establish an explicit representation for the solution of regularized minimization, so that the method can find the global minimizer without being trapped in saddle points, local minima, or discontinuities. The main theoretical result shows that the method enjoys the global linear convergence with high probability for such functions. Besides, the method is derivative-free and its per-iteration cost, i.e., the number of function evaluations, is bounded, so it has a complexity bound $\mathcal{O}(\log\frac{1}{\epsilon})$ for finding a point such that the optimality gap at this point is less than $\epsilon>0$. Numerical experiments in up to $500$ dimensions demonstrate the benefits of the method.
Generalized Derivatives and Optimality Conditions in Nonconvex Optimization

DOI: https://doi.org/10.1007/s40840-024-01676-6

2024-03-26

Bulletin of the Malaysian Mathematical Sciences Society

Abstract:In this paper, we study the radial epiderivative notion for nonconvex functions, which extends the (classical) directional derivative concept. The paper presents new definition and new properties for this notion and establishes relationships between the radial epiderivative, the Clarke's directional derivative, the Rockafellar's subderivative and the directional derivative. The radial epiderivative notion is used to establish new regularity conditions without convexity conditions. The paper presents explicit formulations for computing the radial epiderivatives in terms of weak subgradients and vice versa. We also present an iterative algorithm for approximate computing of radial epiderivatives and show that the algorithm terminates in a finite number of iterations. The paper analyzes necessary and sufficient conditions for global optimums in nonconvex optimization via the radial epiderivatives. We formulate a necessary and sufficient condition for a global descent direction for radially epidifferentiable nonconvex functions. All the properties and theorems presented in this paper are illustrated and interpreted on examples.

mathematics
A descent subgradient method using Mifflin's line search for nonsmooth nonconvex optimization

Morteza Maleknia,Majid Soleimani-damaneh

DOI: https://doi.org/10.1080/02331934.2024.2322152

IF: 2.2

2024-03-03

Optimization

Abstract:We propose a descent subgradient algorithm for minimizing a function f:Rn→R , assumed to be locally Lipschitz, but not necessarily smooth or convex. To find an effective descent direction, the Goldstein ε -subdifferential is approximated through an iterative process. The method enjoys a new two-point variant of Mifflin's line search in which the subgradients are arbitrary. Thus, the line search procedure is easy to implement. Moreover, in comparison to bundle methods, the quadratic subproblems have a simple structure, and to handle nonconvexity the proposed method requires no algorithmic modification. We study the global convergence of the method and prove that any accumulation point of the generated sequence is Clarke stationary, assuming that the objective f is weakly upper semismooth. We illustrate the efficiency and effectiveness of the proposed algorithm on a collection of academic and semi-academic test problems.

mathematics, applied,operations research & management science
Parameter-Free Accelerated Gradient Descent for Nonconvex Minimization

Naoki Marumo,Akiko Takeda

DOI: https://doi.org/10.1137/22m1540934

IF: 2.763

2024-06-19

SIAM Journal on Optimization

Abstract:SIAM Journal on Optimization, Volume 34, Issue 2, Page 2093-2120, June 2024. We propose a new first-order method for minimizing nonconvex functions with a Lipschitz continuous gradient and Hessian. The proposed method is an accelerated gradient descent with two restart mechanisms and finds a solution where the gradient norm is less than [math] in [math] function and gradient evaluations. Unlike existing first-order methods with similar complexity bounds, our algorithm is parameter-free because it requires no prior knowledge of problem-dependent parameters, e.g., the Lipschitz constants and the target accuracy [math]. The main challenge in achieving this advantage is estimating the Lipschitz constant of the Hessian using only first-order information. To this end, we develop a new Hessian-free analysis based on two technical inequalities: a Jensen-type inequality for gradients and an error bound for the trapezoidal rule. Several numerical results illustrate that the proposed method performs comparably to existing algorithms with similar complexity bounds, even without parameter tuning.

mathematics, applied
Walking in the Shadow: A New Perspective on Descent Directions for Constrained Minimization

Hassan Mortagy,Swati Gupta,Sebastian Pokutta

2023-08-31

Abstract:Descent directions such as movement towards Descent directions, including movement towards Frank-Wolfe vertices, away-steps, in-face away-steps and pairwise directions, have been an important design consideration in conditional gradient descent (CGD) variants. In this work, we attempt to demystify the impact of the movement in these directions towards attaining constrained minimizers. The optimal local direction of descent is the directional derivative (i.e., shadow) of the projection of the negative gradient. We show that this direction is the best away-step possible, and the continuous-time dynamics of moving in the shadow is equivalent to the dynamics of projected gradient descent (PGD), although it's non-trivial to discretize. We also show that Frank-Wolfe (FW) vertices correspond to projecting onto the polytope using an "infinite" step in the direction of the negative gradient, thus providing a new perspective on these steps. We combine these insights into a novel Shadow-CG method that uses FW and shadow steps, while enjoying linear convergence, with a rate that depends on the number of breakpoints in its projection curve, rather than the pyramidal width. We provide a linear bound on the number of breakpoints for simple polytopes and present scaling-invariant upper bounds for general polytopes based on the number of facets. We exemplify the benefit of using Shadow-CG computationally for various applications, while raising an open question about tightening the bound on the number of breakpoints for general polytopes.

Optimization and Control,Machine Learning
Universal Gradient Descent Ascent Method for Nonconvex-Nonconcave Minimax Optimization

Taoli Zheng,Linglingzhi Zhu,Anthony Man-Cho So,Jose Blanchet,Jiajin Li

2023-10-30

Abstract:Nonconvex-nonconcave minimax optimization has received intense attention over the last decade due to its broad applications in machine learning. Most existing algorithms rely on one-sided information, such as the convexity (resp. concavity) of the primal (resp. dual) functions, or other specific structures, such as the Polyak-Łojasiewicz (PŁ) and Kurdyka-Łojasiewicz (KŁ) conditions. However, verifying these regularity conditions is challenging in practice. To meet this challenge, we propose a novel universally applicable single-loop algorithm, the doubly smoothed gradient descent ascent method (DS-GDA), which naturally balances the primal and dual updates. That is, DS-GDA with the same hyperparameters is able to uniformly solve nonconvex-concave, convex-nonconcave, and nonconvex-nonconcave problems with one-sided KŁ properties, achieving convergence with $\mathcal{O}(\epsilon^{-4})$ complexity. Sharper (even optimal) iteration complexity can be obtained when the KŁ exponent is known. Specifically, under the one-sided KŁ condition with exponent $\theta\in(0,1)$, DS-GDA converges with an iteration complexity of $\mathcal{O}(\epsilon^{-2\max\{2\theta,1\}})$. They all match the corresponding best results in the literature. Moreover, we show that DS-GDA is practically applicable to general nonconvex-nonconcave problems even without any regularity conditions, such as the PŁ condition, KŁ condition, or weak Minty variational inequalities condition. For various challenging nonconvex-nonconcave examples in the literature, including ``Forsaken'', ``Bilinearly-coupled minimax'', ``Sixth-order polynomial'', and ``PolarGame'', the proposed DS-GDA can all get rid of limit cycles. To the best of our knowledge, this is the first first-order algorithm to achieve convergence on all of these formidable problems.

Optimization and Control,Machine Learning
Dissipative Gradient Descent Ascent Method: A Control Theory Inspired Algorithm for Min-max Optimization

Tianqi Zheng,Nicolas Loizou,Pengcheng You,Enrique Mallada

2024-03-14

Abstract:Gradient Descent Ascent (GDA) methods for min-max optimization problems typically produce oscillatory behavior that can lead to instability, e.g., in bilinear settings. To address this problem, we introduce a dissipation term into the GDA updates to dampen these oscillations. The proposed Dissipative GDA (DGDA) method can be seen as performing standard GDA on a state-augmented and regularized saddle function that does not strictly introduce additional convexity/concavity. We theoretically show the linear convergence of DGDA in the bilinear and strongly convex-strongly concave settings and assess its performance by comparing DGDA with other methods such as GDA, Extra-Gradient (EG), and Optimistic GDA. Our findings demonstrate that DGDA surpasses these methods, achieving superior convergence rates. We support our claims with two numerical examples that showcase DGDA's effectiveness in solving saddle point problems.

Optimization and Control,Machine Learning
First-order optimization on stratified sets

Guillaume Olikier,Kyle A. Gallivan,P.-A. Absil

2023-03-28

Abstract:We consider the problem of minimizing a differentiable function with locally Lipschitz continuous gradient on a stratified set and present a first-order algorithm designed to find a stationary point of that problem. Our assumptions on the stratified set are satisfied notably by the determinantal variety (i.e., matrices of bounded rank), its intersection with the cone of positive-semidefinite matrices, and the set of nonnegative sparse vectors. The iteration map of the proposed algorithm applies a step of projected-projected gradient descent with backtracking line search, as proposed by Schneider and Uschmajew (2015), to its input but also to a projection of the input onto each of the lower strata to which it is considered close, and outputs a point among those thereby produced that maximally reduces the cost function. Under our assumptions on the stratified set, we prove that this algorithm produces a sequence whose accumulation points are stationary, and therefore does not follow the so-called apocalypses described by Levin, Kileel, and Boumal (2022). We illustrate the apocalypse-free property of our method through a numerical experiment on the determinantal variety.

Optimization and Control,Numerical Analysis
Directional proximal point method for convex optimization

Wen-Liang Hwang,Chang-Wei Yueh

2023-12-05

Abstract:The use of proximal point operators for optimization can be computationally expensive when the dimensionality of a function (i.e., the number of variables) is high. In this study, we sought to reduce the cost of calculating proximal point operators by developing a directional operator in which the proximal regularization of a function along a specific direction is penalized. We used this operator in a novel approach to optimization, referred to as the directional proximal point method (Direction PPM). When using Direction PPM, the key to achieving convergence is the selection of direction sequences for directional proximal point operators. In this paper, we present the conditions/assumptions by which to derive directions capable of achieving global convergence for convex functions. Considered a light version of PPM, Direction PPM uses scalar optimization to derive a stable step-size via a direction envelope function and an auxiliary method to derive a direction sequence that satisfies the assumptions. This makes Direction PPM adaptable to a larger class of functions. Through applications to differentiable convex functions, we demonstrate that negative gradient directions at the current iterates could conceivably be used to achieve this end. We provide experimental results to illustrate the efficacy of Direction PPM in practice.

Optimization and Control
On the directional asymptotic approach in optimization theory

Matúš Benko,Patrick Mehlitz

2024-02-26

Abstract:As a starting point of our research, we show that, for a fixed order $\gamma\geq 1$, each local minimizer of a rather general nonsmooth optimization problem in Euclidean spaces is either M-stationary in the classical sense (corresponding to stationarity of order $1$), satisfies stationarity conditions in terms of a coderivative construction of order $\gamma$, or is asymptotically stationary with respect to a critical direction as well as order $\gamma$ in a certain sense. By ruling out the latter case with a constraint qualification not stronger than directional metric subregularity, we end up with new necessary optimality conditions comprising a mixture of limiting variational tools of orders $1$ and $\gamma$. These abstract findings are carved out for the broad class of geometric constraints and $\gamma:=2$, and visualized by examples from complementarity-constrained and nonlinear semidefinite optimization. As a byproduct of the particular setting $\gamma:=1$, our general approach yields new so-called directional asymptotic regularity conditions which serve as constraint qualifications guaranteeing M-stationarity of local minimizers. We compare these new regularity conditions with standard constraint qualifications from nonsmooth optimization. Further, we extend directional concepts of pseudo- and quasi-normality to arbitrary set-valued mappings. It is shown that these properties provide sufficient conditions for the validity of directional asymptotic regularity. Finally, a novel coderivative-like variational tool is used to construct sufficient conditions for the presence of directional asymptotic regularity. For geometric constraints, it is illustrated that all appearing objects can be calculated in terms of initial problem data.

Optimization and Control
Adaptive Algorithms for Relatively Lipschitz Continuous Convex Optimization Problems

Fedor Stonyakin,Alexander Titov,Mohammad Alkousa,Oleg Savchuk,Alexander Gasnikov

2024-04-18

Abstract:Recently there were proposed some innovative convex optimization concepts, namely, relative smoothness [1] and relative strong convexity [2,3]. These approaches have significantly expanded the class of applicability of gradient-type methods with optimal estimates of the convergence rate, which are invariant regardless of the dimensionality of the problem. Later Yu. Nesterov and H. Lu introduced some modifications of the Mirror Descent method for convex minimization problems with the corresponding analogue of the Lipschitz condition (so-called relative Lipschitz continuity). By introducing an artificial inaccuracy to the optimization model, we propose adaptive methods for minimizing a convex Lipschitz continuous function, as well as for the corresponding class of variational inequalities. We also consider an adaptive "universal" method, applicable to convex minimization problems both on the class of relatively smooth and relatively Lipschitz continuous functionals with optimal estimates of the convergence rate. The universality of the method makes it possible to justify the applicability of the obtained theoretical results to a wider class of convex optimization problems. We also present the results of numerical experiments.

Optimization and Control
Orthogonal Directions Constrained Gradient Method: from non-linear equality constraints to Stiefel manifold

Sholom Schechtman,Daniil Tiapkin,Michael Muehlebach,Eric Moulines

2023-03-16

Abstract:We consider the problem of minimizing a non-convex function over a smooth manifold $\mathcal{M}$. We propose a novel algorithm, the Orthogonal Directions Constrained Gradient Method (ODCGM) which only requires computing a projection onto a vector space. ODCGM is infeasible but the iterates are constantly pulled towards the manifold, ensuring the convergence of ODCGM towards $\mathcal{M}$. ODCGM is much simpler to implement than the classical methods which require the computation of a retraction. Moreover, we show that ODCGM exhibits the near-optimal oracle complexities $\mathcal{O}(1/\varepsilon^2)$ and $\mathcal{O}(1/\varepsilon^4)$ in the deterministic and stochastic cases, respectively. Furthermore, we establish that, under an appropriate choice of the projection metric, our method recovers the landing algorithm of Ablin and Peyré (2022), a recently introduced algorithm for optimization over the Stiefel manifold. As a result, we significantly extend the analysis of Ablin and Peyré (2022), establishing near-optimal rates both in deterministic and stochastic frameworks. Finally, we perform numerical experiments which shows the efficiency of ODCGM in a high-dimensional setting.

Optimization and Control,Machine Learning
Saving Gradient and Negative Curvature Computations: Finding Local Minima More Efficiently

Yaodong Yu,Difan Zou,Quanquan Gu

DOI: https://doi.org/10.48550/arXiv.1712.03950

IF: 5.414

2017-12-11

Machine Learning

Abstract:We propose a family of nonconvex optimization algorithms that are able to save gradient and negative curvature computations to a large extent, and are guaranteed to find an approximate local minimum with improved runtime complexity. At the core of our algorithms is the division of the entire domain of the objective function into small and large gradient regions: our algorithms only perform gradient descent based procedure in the large gradient region, and only perform negative curvature descent in the small gradient region. Our novel analysis shows that the proposed algorithms can escape the small gradient region in only one negative curvature descent step whenever they enter it, and thus they only need to perform at most $N_{\epsilon}$ negative curvature direction computations, where $N_{\epsilon}$ is the number of times the algorithms enter small gradient regions. For both deterministic and stochastic settings, we show that the proposed algorithms can potentially beat the state-of-the-art local minima finding algorithms. For the finite-sum setting, our algorithm can also outperform the best algorithm in a certain regime.
Regularized asymptotic descents: finding the global minima for a class of multiple minima problems

Xiaopeng Luo,Xin Xu

2020-01-01

Abstract:We propose and analyze regularized asymptotic descent (RAD) methods to find the global minimizer for a class of nonconvex, nonsmooth, or even discontinuous multiple minima functions that are extended from strongly convex functions with Lipschitz-continuous gradients. First, we establish an asymptotic representation for the solution of regularized minimization, then we construct the method based on this representation. The method can find the global minimizer without being trapped in saddle points, local minima, or discontinuities. The main theoretical result shows that the method enjoys the global linear convergence for such a class of functions. Furthermore, the method is derivative-free and its per-iteration cost, i.e., the number of function evaluations, is also bounded, so it has a complexity bound $\mathcal{O}(\log\frac{1}{\epsilon})$ for finding a point such that the gap between this point and the global minimizer is less than $\epsilon>0$. Numerical experiments in up to $500$ dimensions demonstrate the benefits of the method.

Directional descent

Directional Smoothness and Gradient Methods: Convergence and Adaptivity

Derivation of Coordinate Descent Algorithms from Optimal Control Theory

A Descent Method for Nonsmooth Multiobjective Optimization in Hilbert Spaces

A descent subgradient method using Mifflin line search for nonsmooth nonconvex optimization

Regularized Asymptotic Descents for a Class of Nonconvex Optimization Problems.

Generalized Derivatives and Optimality Conditions in Nonconvex Optimization

A descent subgradient method using Mifflin's line search for nonsmooth nonconvex optimization

Parameter-Free Accelerated Gradient Descent for Nonconvex Minimization

Walking in the Shadow: A New Perspective on Descent Directions for Constrained Minimization

Universal Gradient Descent Ascent Method for Nonconvex-Nonconcave Minimax Optimization

Dissipative Gradient Descent Ascent Method: A Control Theory Inspired Algorithm for Min-max Optimization

First-order optimization on stratified sets

Directional proximal point method for convex optimization

On the directional asymptotic approach in optimization theory

Adaptive Algorithms for Relatively Lipschitz Continuous Convex Optimization Problems

Orthogonal Directions Constrained Gradient Method: from non-linear equality constraints to Stiefel manifold

Saving Gradient and Negative Curvature Computations: Finding Local Minima More Efficiently

Regularized asymptotic descents: finding the global minima for a class of multiple minima problems