Convergence analysis of the Fast Subspace Descent method for convex optimization problems

Long Chen,Xiaozhe Hu,Steven M. Wise

DOI: https://doi.org/10.1090/mcom/3526

IF: 2.118

2020-04-07

Mathematics of Computation

Abstract:Long Chen, Xiaozhe Hu and Steven M. Wise Math. Comp. 89 (2020), 2249-2282. , references and article information

mathematics, applied

What problem does this paper attempt to address?

Convergence Analysis of the Fast Subspace Descent Methods for Convex Optimization Problems

Long Chen,Xiaozhe Hu,Steven M. Wise

DOI: https://doi.org/10.48550/arXiv.1810.04116

2019-10-20

Abstract:The full approximation storage (FAS) scheme is a widely used multigrid method for nonlinear problems. In this paper, a new framework to design and analyze FAS-like schemes for convex optimization problems is developed. The new method, the Fast Subspace Descent (FASD) scheme, which generalizes classical FAS, can be recast as an inexact version of nonlinear multigrid methods based on space decomposition and subspace correction. The local problem in each subspace can be simplified to be linear and one gradient descent iteration (with an appropriate step size) is enough to ensure a global linear (geometric) convergence of FASD.

Numerical Analysis
A GLOBALLY AND SUPERLIEARLY CONVERGENT FEASIBLE SQP ALGORITHM FOR DEGENERATE CONSTRAINED OPTIMIZATION

Yu Chen,Xie Xiaoliang

2010-01-01

Journal of applied mathematics & informatics

Abstract:In this paper, A FSQP algorithm for degenerate inequality constraints optimization problems is proposed. At each iteration of the proposed algorithm, a feasible direction of descent is obtained by solving a quadratic programming subproblem. To overcome the Maratos effect, a higher-order correction direction is obtained by solving another quadratic programming subproblem. The algorithm is proved to be globally convergent and superlinearly convergent under some mild conditions. Finally, some preliminary numerical results are reported.
Convergence of Inexact Steepest Descent Algorithm for Multiobjective Optimizations on Riemannian Manifolds Without Curvature Constraints

X. M. Wang,J. H. Wang,C. Li

DOI: https://doi.org/10.1007/s10957-023-02235-y

2023-01-01

Abstract:We study the issue of convergence for inexact steepest descent algorithm (employing general step sizes) for multiobjective optimizations on general Riemannian manifolds (without curvature constraints). Under the assumption of the local convexity/quasi-convexity, local/global convergence results are established. Furthermore, without the assumption of the local convexity/quasi-convexity, but under an error bound-like condition, local/global convergence results and convergence rate estimates are presented, which are new even in the linear space setting. Our results improve/extend the corresponding ones in (Wang et al. in SIAM J Optim 31(1):172–199, 2021) for scalar optimization problems on Riemannian manifolds to multiobjective ones. Finally, for the special case when the inexact steepest descent algorithm employing Armijo rule, our results improve/extend the corresponding ones in (Ferreira et al. in J Optim Theory Appl 184:507–533, 2020) by relaxing curvature constraints.
On Convergence of the Gauss-Newton Method for Convex Composite Optimization

Chong Li,Xinghua Wang

DOI: https://doi.org/10.1007/s101070100249

IF: 2.7

2002-01-01

Mathematical Programming

Abstract:The local quadratic convergence of the Gauss-Newton method for convex composite optimization f=h∘F is established for any convex function h with the minima set C, extending Burke and Ferris’ results in the case when C is a set of weak sharp minima for h.
Descent Methods for Vector Optimization Problems: A Majorization-Minimization Perspective

Jian Chen,Liping Tang,Xinmin Yang

2024-07-18

Abstract:In this paper, we develop a unified framework and convergence analysis of descent methods for vector optimization problems (VOPs) from a majorization-minimization perspective. By choosing different surrogate functions, the generic method reduces to some existing descent methods with and without line search, respectively. The unified convergence analysis shows that the slow convergence of the steepest descent method is mainly due to the large gap between surrogate and objective functions. As a result, the performance of descent methods can be improved by narrowing the surrogate gap. Interestingly, we observe that selecting a tighter surrogate function is equivalent to using an appropriate base of the dual cone in the direction-finding subproblem. Furthermore, we use Barzilai-Borwein method to narrow the surrogate gap and devise a Barzilai-Borwein descent method for VOPs with polyhedral cone. By reformulating the subproblem, we provide a new insight into the Barzilai-Borwein descent method and bridge it to the steepest descent method. Finally, several numerical experiments confirm the efficiency of the proposed method.

Optimization and Control
Subspace Methods for Nonlinear Optimization

Xin Liu,Zaiwen Wen,Ya-Xiang Yuan

DOI: https://doi.org/10.4208/csiam-am.so-2021-0016

2021-01-01

CSIAM Transactions on Applied Mathematics

Abstract:Subspace techniques such as Krylov subspace methods have been well known and extensively used in numerical linear algebra. They are also ubiquitous and becoming indispensable tools in nonlinear optimization due to their ability to handle large scale problems. There are generally two types of principals: i) the decision variable is updated in a lower dimensional subspace; ii) the objective function or constraints are approximated in a certain smaller subspace of their domain. The key ingredients are the constructions of suitable subspaces and subproblems according to the specific structures of the variables and functions such that either the exact or inexact solutions of subproblems are readily available and the corresponding computational cost is significantly reduced. A few relevant techniques include but not limited to direct combinations, block coordinate descent, active sets, limited-memory, Anderson acceleration, subspace correction, sampling and sketching. This paper gives a comprehensive survey on the subspace methods and their recipes in unconstrained and constrained optimization, nonlinear least squares problem, sparse and low rank optimization, linear and nonlinear eigenvalue computation, semidefinite programming, stochastic optimization and etc. In order to provide helpful guidelines, we emphasize on high level concepts for the development and implementation of practical algorithms from the subspace framework.
Convergence rate analysis of the gradient descent–ascent method for convex–concave saddle-point problems

Moslem Zamani,Hadi Abbaszadehpeivasti,Etienne de Klerk

DOI: https://doi.org/10.1080/10556788.2024.2360040

2024-06-22

Optimization Methods and Software

Abstract:In this paper, we study the gradient descent–ascent method for convex–concave saddle-point problems. We derive a new non-asymptotic global convergence rate in terms of distance to the solution set by using the semidefinite programming performance estimation method. The given convergence rate incorporates most parameters of the problem and it is exact for a large class of strongly convex-strongly concave saddle-point problems for one iteration. We also investigate the algorithm without strong convexity and we provide some necessary and sufficient conditions under which the gradient descent–ascent enjoys linear convergence.

operations research & management science,mathematics, applied,computer science, software engineering
Convergence rates analysis of Interior Bregman Gradient Method for Vector Optimization Problems

Jian Chen,Liping Tang,Xinmin Yang

DOI: https://doi.org/10.48550/arXiv.2206.10070

2022-06-21

Optimization and Control

Abstract:In recent years, by using Bregman distance, the Lipschitz gradient continuity and strong convexity were lifted and replaced by relative smoothness and relative strong convexity. Under the mild assumptions, it was proved that gradient methods with Bregman regularity converge linearly for single-objective optimization problems (SOPs). In this paper, we extend the relative smoothness and relative strong convexity to vector-valued functions and analyze the convergence of an interior Bregman gradient method for vector optimization problems (VOPs). Specifically, the global convergence rates are $\mathcal{O}(\frac{1}{k})$ and $\mathcal{O}(r^{k})(0<r<1)$ for convex and relative strongly convex VOPs, respectively. Moreover, the proposed method converges linearly for VOPs that satisfy a vector Bregman-PL inequality.
Faster Accelerated First-order Methods for Convex Optimization with Strongly Convex Function Constraints

Zhenwei Lin,Qi Deng

2024-11-27

Abstract:In this paper, we introduce faster accelerated primal-dual algorithms for minimizing a convex function subject to strongly convex function constraints. Prior to our work, the best complexity bound was $\mathcal{O}(1/{\varepsilon})$, regardless of the strong convexity of the constraint function. It is unclear whether the strong convexity assumption can enable even better convergence results. To address this issue, we have developed novel techniques to progressively estimate the strong convexity of the Lagrangian function. Our approach, for the first time, effectively leverages the constraint strong convexity, obtaining an improved complexity of $\mathcal{O}(1/\sqrt{\varepsilon})$. This rate matches the complexity lower bound for strongly-convex-concave saddle point optimization and is therefore order-optimal. We show the superior performance of our methods in sparsity-inducing constrained optimization, notably Google's personalized PageRank problem. Furthermore, we show that a restarted version of the proposed methods can effectively identify the optimal solution's sparsity pattern within a finite number of steps, a result that appears to have independent significance.

Optimization and Control,Machine Learning
A General Two-Level Subspace Method for Nonlinear Optimization

Cheng Chen,Zaiwen Wen,Ya-xiang Yuan

DOI: https://doi.org/10.4208/jcm.1706-m2016-0721

2018-01-01

Journal of Computational Mathematics

Abstract:A new two-level subspace method is proposed for solving the general unconstrained minimization formulations discretized from infinite-dimensional optimization problems. At each iteration, the algorithm executes either a direct step on the current level or a coarse subspace correction step. In the coarse subspace correction step, we augment the traditional coarse grid space by a two-dimensional subspace spanned by the coordinate direction and the gradient direction at the current point. Global convergence is proved and convergence rate is studied under some mild conditions on the discretized functions. Preliminary numerical experiments on a few variational problems show that our two-level subspace method is promising.
On Some Versions of Subspace Optimization Methods with Inexact Gradient Information

Ilya Kuruzov,Fedor Stonyakin

2024-07-01

Abstract:It is well-known that accelerated gradient first order methods possess optimal complexity estimates for the class of convex smooth minimization problems. In many practical situations, it makes sense to work with inexact gradients. However, this can lead to the accumulation of corresponding inexactness in the theoretical estimates of the rate of convergence. We propose some modification of the methods for convex optimization with inexact gradient based on the subspace optimization such as Nemirovski's Conjugate Gradients and Sequential Subspace Optimization. We research the method convergence for different condition of inexactness both in gradient value and accuracy of subspace optimization problems. Besides this, we investigate generalization of this result to the class of quasar-convex (weakly-quasi-convex) functions.

Optimization and Control
A Descent Method for Nonsmooth Multiobjective Optimization in Hilbert Spaces

Konstantin Sonntag,Bennet Gebken,Georg Müller,Sebastian Peitz,Stefan Volkwein

DOI: https://doi.org/10.1007/s10957-024-02520-4

2024-09-14

Journal of Optimization Theory and Applications

Abstract:The efficient optimization method for locally Lipschitz continuous multiobjective optimization problems from Gebken and Peitz (J Optim Theory Appl 188:696–723, 2021) is extended from finite-dimensional problems to general Hilbert spaces. The method iteratively computes Pareto critical points, where in each iteration, an approximation of the Clarke subdifferential is computed in an efficient manner and then used to compute a common descent direction for all objective functions. To prove convergence, we present some new optimality results for nonsmooth multiobjective optimization problems in Hilbert spaces. Using these, we can show that every accumulation point of the sequence generated by our algorithm is Pareto critical under common assumptions. Computational efficiency for finding Pareto critical points is numerically demonstrated for multiobjective optimal control of an obstacle problem.

mathematics, applied,operations research & management science
Fast Subspace Optimization Method for Nonlinear Inverse Problems in Banach Spaces with Uniformly Convex Penalty Terms

Ruixue Gu,Bo Han,Yong Chen

DOI: https://doi.org/10.1088/1361-6420/ab2a2b

IF: 2.408

2019-01-01

Inverse Problems

Abstract:In this paper, we consider the subspace optimization method for solving nonlinear inverse problems in Banach spaces, which is based on the sequential Bregman projections with uniformly convex penalty term. The penalty term is allowed to be non-smooth, including L-1 and total variation like penalty functionals, to reconstruct the special features of solutions such as sparsity and discontinuities. Instead of just utilizing the current gradient like the Landweber iteration, the method uses multiple search directions in each iteration to accelerate the convergence. Moreover, their step lengths are calculated by the projection onto the subspace that contains the solution set of the unperturbed problem. Under certain assumptions, we present the detailed convergence analysis when the data is given exactly. For the data containing noise, we use the discrepancy principle as stopping rule and then obtain the regularization result of the method. Finally, some numerical simulations for parameter identification problems are provided to illustrate the effectiveness of capturing the property of exact solutions and the acceleration effect of the method.
Geodesic Convexity of the Symmetric Eigenvalue Problem and Convergence of Steepest Descent

Foivos Alimisis,Bart Vandereycken

DOI: https://doi.org/10.1007/s10957-024-02538-8

2024-10-11

Journal of Optimization Theory and Applications

Abstract:We study the convergence of the Riemannian steepest descent algorithm on the Grassmann manifold for minimizing the block version of the Rayleigh quotient of a symmetric matrix. Even though this problem is non-convex in the Euclidean sense and only very locally convex in the Riemannian sense, we discover a structure for this problem that is similar to geodesic strong convexity, namely, weak-strong convexity. This allows us to apply similar arguments from convex optimization when studying the convergence of the steepest descent algorithm but with initialization conditions that do not depend on the eigengap . When , we prove exponential convergence rates, while otherwise the convergence is algebraic. Additionally, we prove that this problem is geodesically convex in a neighbourhood of the global minimizer of radius .

mathematics, applied,operations research & management science
Provably Faster Gradient Descent via Long Steps

Benjamin Grimmer

DOI: https://doi.org/10.1137/23m1588408

IF: 2.763

2024-07-18

SIAM Journal on Optimization

Abstract:SIAM Journal on Optimization, Volume 34, Issue 3, Page 2588-2608, September 2024. This work establishes new convergence guarantees for gradient descent in smooth convex optimization via a computer-assisted analysis technique. Our theory allows nonconstant stepsize policies with frequent long steps potentially violating descent by analyzing the overall effect of many iterations at once rather than the typical one-iteration inductions used in most first-order method analyses. We show that long steps, which may increase the objective value in the short term, lead to provably faster convergence in the long term. A conjecture towards proving a faster [math] rate for gradient descent is also motivated along with simple numerical validation.

mathematics, applied
A Universal Accelerated Primal–Dual Method for Convex Optimization Problems

Hao Luo

DOI: https://doi.org/10.1007/s10957-024-02394-6

2024-03-03

Journal of Optimization Theory and Applications

Abstract:This work presents a universal accelerated primal–dual method for affinely constrained convex optimization problems. It can handle both Lipschitz and Hölder gradients but does not need to know the smoothness level of the objective function. In line search part, it uses dynamically decreasing parameters and produces approximate Lipschitz constant with moderate magnitude. In addition, based on a suitable discrete Lyapunov function and tight decay estimates of some differential/difference inequalities, a universal optimal mixed-type convergence rate is established. Some numerical tests are provided to confirm the efficiency of the proposed method.

mathematics, applied,operations research & management science
Analyzing the speed of convergence in nonsmooth optimization via the Goldstein subdifferential with application to descent methods

Bennet Gebken

2024-10-02

Abstract:The Goldstein $\varepsilon$-subdifferential is a relaxed version of the Clarke subdifferential which has recently appeared in several algorithms for nonsmooth optimization. With it comes the notion of $(\varepsilon,\delta)$-critical points, which are points in which the element with the smallest norm in the $\varepsilon$-subdifferential has norm at most $\delta$. To obtain points that are critical in the classical sense, $\varepsilon$ and $\delta$ must vanish. In this article, we analyze at which speed the distance of $(\varepsilon,\delta)$-critical points to the minimum vanishes with respect to $\varepsilon$ and $\delta$. Afterwards, we apply our results to gradient sampling methods and perform numerical experiments. Throughout the article, we put a special emphasis on supporting the theoretical results with simple examples that visualize them.

Optimization and Control
Efficient Accelerated Coordinate Descent Methods and Faster Algorithms for Solving Linear Systems

Yin Tat Lee,Aaron Sidford

DOI: https://doi.org/10.1109/focs.2013.24

2013-10-01

Abstract:In this paper we show how to accelerate randomized coordinate descent methods and achieve faster convergence rates without paying per-iteration costs in asymptotic running time. In particular, we show how to generalize and efficiently implement a method proposed by Nesterov, giving faster asymptotic running times for various algorithms that use standard coordinate descent as a black box. In addition to providing a proof of convergence for this new general method, we show that it is numerically stable, efficiently implementable, and in certain regimes, asymptotically optimal. To highlight the power of this algorithm, we show how it can used to create faster linear system solvers in several regimes: • We show how this method achieves a faster asymptotic runtime than conjugate gradient for solving a broad class of symmetric positive definite systems of equations. • We improve the convergence guarantees for Kaczmarz methods, a popular technique for image reconstruction and solving overdetermined systems of equations, by accelerating an algorithm of Strohmer and Vershynin. • We achieve the best known running time for solving Symmetric Diagonally Dominant (SDD) system of equations in the unit-cost RAM model, obtaining a running time of ${\bf O}({\bf m}\log^{\wedge}\{3/2\}{\bf n}(\log\log {\bf n})^{\wedge}\{1/2\}\log((\log {\bf n})/{\bf eps}))$ by accelerating a recent solver by Kelner et al. We show how this method achieves a faster asymptotic runtime than conjugate gradient for solving a broad class of symmetric positive definite systems of equations. We improve the convergence guarantees for Kaczmarz methods, a popular technique for image reconstruction and solving overdetermined systems of equations, by accelerating an algorithm of Strohmer and Vershynin. We achieve the best known running time for solving Symmetric Diagonally Dominant (SDD) system of equations in the unit-cost RAM model, obtaining a running time of ${\bf O}({\bf m}\log^{\wedge}\{3/2\}{\bf n}(\log\log {\bf n})^{\wedge}\{1/2\}\log((\log {\bf n})/{\bf eps}))$ by accelerating a recent solver by Kelner et al. Beyond the independent interest of these solvers, we believe they highlight the versatility of the approach of this paper and we hope that they will open the door for further algorithmic improvements in the future.
A Gradient Complexity Analysis for Minimizing the Sum of Strongly Convex Functions with Varying Condition Numbers

Nuozhou Wang,Shuzhong Zhang

DOI: https://doi.org/10.1137/22m1503646

IF: 2.763

2024-04-12

SIAM Journal on Optimization

Abstract:SIAM Journal on Optimization, Volume 34, Issue 2, Page 1374-1401, June 2024. A popular approach to minimizing a finite sum of smooth convex functions is stochastic gradient descent (SGD) and its variants. Fundamental research questions associated with SGD include (i) how to find a lower bound on the number of times that the gradient oracle of each individual function must be assessed in order to find an [math]-minimizer of the overall objective; (ii) how to design algorithms which guarantee finding an [math]-minimizer of the overall objective in expectation no more than a certain number of times (in terms of [math]) that the gradient oracle of each function needs to be assessed (i.e., upper bound). If these two bounds are at the same order of magnitude, then the algorithms may be called optimal. Most existing results along this line of research typically assume that the functions in the objective share the same condition number. In this paper, the first model we study is the problem of minimizing the sum of finitely many strongly convex functions whose condition numbers are all different. We propose an SGD-based method for this model and show that it is optimal in gradient computations, up to a logarithmic factor. We then consider a constrained separate block optimization model and present lower and upper bounds for its gradient computation complexity. Next, we propose solving the Fenchel dual of the constrained block optimization model via generalized SSNM, which we introduce earlier, and show that it yields a lower iteration complexity than solving the original model by the ADMM-type approach. Finally, we extend the analysis to the general composite convex optimization model and obtain gradient-computation complexity results under certain conditions.

mathematics, applied

Convergence analysis of the Fast Subspace Descent method for convex optimization problems

Convergence Analysis of the Fast Subspace Descent Methods for Convex Optimization Problems

A GLOBALLY AND SUPERLIEARLY CONVERGENT FEASIBLE SQP ALGORITHM FOR DEGENERATE CONSTRAINED OPTIMIZATION

Convergence of Inexact Steepest Descent Algorithm for Multiobjective Optimizations on Riemannian Manifolds Without Curvature Constraints

On Convergence of the Gauss-Newton Method for Convex Composite Optimization

Descent Methods for Vector Optimization Problems: A Majorization-Minimization Perspective

Subspace Methods for Nonlinear Optimization

Convergence rate analysis of the gradient descent–ascent method for convex–concave saddle-point problems

Convergence rates analysis of Interior Bregman Gradient Method for Vector Optimization Problems

Faster Accelerated First-order Methods for Convex Optimization with Strongly Convex Function Constraints

A General Two-Level Subspace Method for Nonlinear Optimization

On Some Versions of Subspace Optimization Methods with Inexact Gradient Information

A Descent Method for Nonsmooth Multiobjective Optimization in Hilbert Spaces

Fast Subspace Optimization Method for Nonlinear Inverse Problems in Banach Spaces with Uniformly Convex Penalty Terms

Geodesic Convexity of the Symmetric Eigenvalue Problem and Convergence of Steepest Descent

Provably Faster Gradient Descent via Long Steps

A Universal Accelerated Primal–Dual Method for Convex Optimization Problems

Analyzing the speed of convergence in nonsmooth optimization via the Goldstein subdifferential with application to descent methods

Efficient Accelerated Coordinate Descent Methods and Faster Algorithms for Solving Linear Systems

A Gradient Complexity Analysis for Minimizing the Sum of Strongly Convex Functions with Varying Condition Numbers