The Mgaor-Iterative Method for Saddle Point Problems

Jicheng Li,Yong Gao,Ziya Mei

2012-01-01

Abstract:In this paper, we explore the solution of saddle point problems and a new iterative method, named modified generalized accelerated overrelaxation (MGAOR) iterative method, is presented. The proposed MGAOR-iterative method is also a generalization of the SOR-like method, and we also analyze the convergence of the MGAOR-iterative method.

What problem does this paper attempt to address?

On the Iterative Algorithm for Large Sparse Saddle Point Problems

Xiaofeng Ling,Xiaozhe Hu

DOI: https://doi.org/10.1016/j.amc.2005.11.052

IF: 4.397

2006-01-01

Applied Mathematics and Computation

Abstract:In this paper, we discuss some iterated method for solving the saddle point problem. We propose some new schemes and prove its convergence. The method has weaker convergence condition than the classic Uzawa method. The analysis is supported by numerical experiments.
An Inexact Uzawa-Type Iterative Method For Solving Saddle Point Problems

Xiao-liang Cheng,Jun Zou

DOI: https://doi.org/10.1080/00207160304658

IF: 1.75

2003-01-01

International Journal of Computer Mathematics

Abstract:An inexact Uzawa-type iterative algorithm is proposed for solving saddle point problems arising from mixed finite element methods or Lagrange multiplier methods for some PDEs. The convergence rate is given in terms of the rates of the two basic iterations and it is shown that the algorithm always converges as long as the two basic iterations converge.
A preconditioned iteration method for solving saddle point problems

Juan Zhang,Yiyi Luo

2024-04-09

Abstract:This paper introduces a preconditioned method designed to comprehensively address the saddle point system with the aim of improving convergence efficiency. In the preprocessor construction phase, a technical approach for solving the approximate inverse matrix of sparse matrices is presented. The effectiveness of the proposed method is demonstrated through numerical examples, emphasizing its efficacy in approximating the inverse matrix. Furthermore, the preprocessing technology includes a low-rank processing step, effectively reducing algorithmic complexity. Numerical experiments validate the effectiveness and feasibility of PSLR-GMRES in solving the saddle point system.

Numerical Analysis
A special parameterized inexact Uzawa algorithm for symmetric saddle point problem

Jun-Feng Lu,Li Ma

DOI: https://doi.org/10.2298/TSCI1804715L

IF: 1.971

2018-01-01

Thermal Science

Abstract:In this paper, we consider a symmetric saddle point problem arising in the fluid dynamics. A special parameterized inexact Uzawa algorithm is proposed for solving the symmetric saddle point problem. The convergence of this special algorithm is considered. Sufficient conditions for the convergence are given. Numerical experiments resulting from stokes problem are presented to show the efficiency of the algorithm.
Multiple Greedy Quasi-Newton Methods for Saddle Point Problems

Minheng Xiao,Shi Bo,Zhizhong Wu

2024-08-01

Abstract:This paper introduces the Multiple Greedy Quasi-Newton (MGSR1-SP) method, a novel approach to solving strongly-convex-strongly-concave (SCSC) saddle point problems. Our method enhances the approximation of the squared indefinite Hessian matrix inherent in these problems, significantly improving both stability and efficiency through iterative greedy updates. We provide a thorough theoretical analysis of MGSR1-SP, demonstrating its linear-quadratic convergence rate. Numerical experiments conducted on AUC maximization and adversarial debiasing problems, compared with state-of-the-art algorithms, underscore our method's enhanced convergence rate. These results affirm the potential of MGSR1-SP to improve performance across a broad spectrum of machine learning applications where efficient and accurate Hessian approximations are crucial.

Artificial Intelligence
On Semi-Convergence of the Generalized Shift-Splitting Iteration Method for Singular Nonsymmetric Saddle Point Problems

Yang Cao,Shu-Xin Miao

DOI: https://doi.org/10.1016/j.camwa.2016.02.027

IF: 3.218

2016-01-01

Computers & Mathematics with Applications

Abstract:Recently, a class of generalized shift-splitting iteration methods were proposed for solving nonsingular nonsymmetric saddle point problems (Cao et al., 2015). In this paper, the generalized shift-splitting iteration method is extended to solve singular nonsymmetric saddle point problems. Semi-convergence of the generalized shift-splitting iteration method is carefully analyzed, which shows that the iterative sequence generated by the generalized shift-splitting iteration method converges to a solution of the singular saddle point problem unconditionally. Numerical results verify the robustness and efficiency of the generalized shift-splitting iteration method.
A Stochastic Proximal Point Algorithm for Saddle-Point Problems

Luo Luo,Chen Cheng,Li Yujun,Xie Guangzeng,Zhang Zhihua

2019-01-01

Abstract: We consider saddle point problems which objective functions are the average of $n$ strongly convex-concave individual components. Recently, researchers exploit variance reduction methods to solve such problems and achieve linear-convergence guarantees. However, these methods have a slow convergence when the condition number of the problem is very large. In this paper, we propose a stochastic proximal point algorithm, which accelerates the variance reduction method SAGA for saddle point problems. Compared with the catalyst framework, our algorithm reduces a logarithmic term of condition number for the iteration complexity. We adopt our algorithm to policy evaluation and the empirical results show that our method is much more efficient than state-of-the-art methods.
An Iterative Minimization Formulation for Saddle Point Search.

Weiguo Gao,Jing Leng,Xiang Zhou

DOI: https://doi.org/10.1137/130930339

IF: 3.039

2015-01-01

SIAM Journal on Numerical Analysis

Abstract:This paper proposes and analyzes an iterative minimization formulation for searching index-1 saddle points of an energy function. We give a general and rigorous description of eigenvector-following methodology in this iterative scheme by considering an auxiliary optimization problem at each iteration in which the new objective function is locally defined near the current guess. We prove that this scheme has a quadratic local convergence rate in terms of number of iterations, in comparison to the linear rate of the gentlest ascent dynamics [W. E and X. Zhou, Nonlinearity, 24 (2011), pp. 1831-1842] and many other existing methods. We also propose the generalization of the new methodology for saddle points of higher index and for constrained energy functions on the manifold. Preliminary numerical results on the nature of this iterative minimization formulation are presented.
Generalized Optimistic Methods for Convex-Concave Saddle Point Problems

Ruichen Jiang,Aryan Mokhtari

2024-01-10

Abstract:The optimistic gradient method has seen increasing popularity for solving convex-concave saddle point problems. To analyze its iteration complexity, a recent work [<a class="link-https" data-arxiv-id="1906.01115" href="https://arxiv.org/abs/1906.01115">arXiv:1906.01115</a>] proposed an interesting perspective that interprets this method as an approximation to the proximal point method. In this paper, we follow this approach and distill the underlying idea of optimism to propose a generalized optimistic method, which includes the optimistic gradient method as a special case. Our general framework can handle constrained saddle point problems with composite objective functions and can work with arbitrary norms using Bregman distances. Moreover, we develop a backtracking line search scheme to select the step sizes without knowledge of the smoothness coefficients. We instantiate our method with first-, second- and higher-order oracles and give best-known global iteration complexity bounds. For our first-order method, we show that the averaged iterates converge at a rate of $O(1/N)$ when the objective function is convex-concave, and it achieves linear convergence when the objective is strongly-convex-strongly-concave. For our second- and higher-order methods, under the additional assumption that the distance-generating function has Lipschitz gradient, we prove a complexity bound of $O(1/\epsilon^\frac{2}{p+1})$ in the convex-concave setting and a complexity bound of $O((L_pD^\frac{p-1}{2}/\mu)^\frac{2}{p+1}+\log\log\frac{1}{\epsilon})$ in the strongly-convex-strongly-concave setting, where $L_p$ ($p\geq 2$) is the Lipschitz constant of the $p$-th-order derivative, $\mu$ is the strong convexity parameter, and $D$ is the initial Bregman distance to the saddle point. Moreover, our line search scheme provably only requires a constant number of calls to a subproblem solver per iteration on average, making our first- and second-order methods particularly amenable to implementation.

Optimization and Control,Machine Learning
On Preconditioned Generalized Shift-Splitting Iteration Methods for Saddle Point Problems.

Yang Cao,Shu-Xin Miao,Zhi-Ru Ren

DOI: https://doi.org/10.1016/j.camwa.2017.05.031

IF: 3.218

2017-01-01

Computers & Mathematics with Applications

Abstract:We study a preconditioned generalized shift-splitting iteration method for solving saddle point problems. The unconditional convergence theory of the preconditioned generalized shift-splitting iteration method is established. When the splitting matrix is used as a preconditioner, we analyze eigenvalue distribution of the preconditioned saddle point matrix. It is proved that complex eigenvalues having nonzero imaginary parts of the preconditioned matrix are located in an intersection of two circles and the real parts of all eigenvalues of the preconditioned matrix are located in a positive interval. Numerical experiments are used to verify our theoretical results and illustrate effectiveness of the proposed iteration method and the corresponding splitting preconditioner.
The Polynomial Acceleration of GSOR Iterative Algorithms for Large Sparse Saddle Point Problems

Pan Chun-ping,Hongyu Wang

2011-01-01

Abstract:In order to solve large sparse saddle point problems(SPP) quickly and efficiently,we construct in this paper a polynomial accelerative iterative algorithm through accelerating the generalized SOR(GSOR) iterative algorithm and using the Chebyshev polynomial.The convergence of the algorithm is also studied.Meanwhile,by examining the convergence of accelerated iterative matrix,the result shows that the new method converges faster than the GSOR method.Numerical results further demonstrate that the new method is more efficient than the GSOR method.
A Decentralized Proximal Point-type Method for Non-convex Non-concave Saddle Point Problems

Weijie Liu,Aryan Mokhtari,Asuman Ozdaglar,Sarath Pattathil,Zebang Shen,Nenggan Zheng

2020-01-01

Abstract:In this paper, we focus on solving a class of constrained non-convex non-concave saddle point problems in a decentralized manner by a group of nodes in a network. Specifically, we assume that each node has access to a summand of a global objective function and nodes are allowed to exchange information only with their neighboring nodes. We propose a decentralized variant of the proximal point method for solving this problem. We show that when the objective function is ρ-weakly convex-weakly concave the iterates converge to approximate stationarity with a rate of O(1/ √ T ) where the approximation error depends linearly on √ ρ. We further show that when the objective function satisfies the Minty VI condition (which generalizes the convex-concave case) we obtain convergence to stationarity with a rate of O(1/ √ T ). To the best of our knowledge, our proposed method is the first decentralized algorithm with theoretical guarantees for solving a non-convex non-concave decentralized saddle point problem.
Some Remarks on the Nonlinear Inexact Uzawa Algorithm for Saddle Point Problem

Haifeng Ye,Xiao-Liang Cheng

DOI: https://doi.org/10.1080/00207160412331284114

2005-01-01

Abstract:In this paper, we discuss and analyze a slight modification of the nonlinear inexact Uzawa algorithm proposed by Bramble et al.. [Bramble, J. H., Pasciak, E. and Vassilev, A. T. (1997). Analysis of the inexact Uzawa algorithm for saddle point problems. SIAM J. Numer. Anal., 34, 1072–1092.] for solving saddle point problems. It is shown that the modified algorithm converges under a condition weaker than that given by Bramble et al. [Bramble, J. H., Pasciak, E. and Vassilev, A. T. (1997). Analysis of the inexact Uzawa algorithm for saddle point problems. SIAM J. Numer. Anal., 34, 1072–1092.] and Cheng [Cheng, X. L. (2000). On the nonlinear inexact Uzawa algorithm for saddle-point problems. SIAM J. Numer. Anal., 37, 1930–1934.]. In addition, we also compare the scheme to the iterative method with variable relaxation parameters proposed by Hu and Zou [Hu, Q. Y. and Zou, J. (2001). An iterative method with variable relaxation parameters for saddle-point problems. SIAM J. Matrix Anal. Appl., 23, 317–338.]. We remove the assumptions presented by Hu and Zou [Hu, Q. Y. and Zou, J. (2001). An iterative method with variable relaxation parameters for saddle-point problems. SIAM J. Matrix Anal. Appl., 23, 317–338.] for the preconditioner Q A .
Preconditioned iterative method for nonsymmetric saddle point linear systems

Li-Dan Liao,Guo-Feng Zhang,Xiang Wang

DOI: https://doi.org/10.1016/j.camwa.2021.07.002

2021-09-01

Abstract:<p>In this paper, a new preconditioned iterative method is presented to solve a class of nonsymmetric nonsingular or singular saddle point problems. The implementation of the proposed preconditioned Krylov subspace method avoids solving inverse of Schur complement and only needs to solve one linear sub-system at each step, which implies that it may save considerable costs. Theoretical convergence analysis, including the bounds of eigenvalues and eigenvectors, the degree of the minimal polynomial of the preconditioned matrix, are discussed in details. Moreover, a novel algebraic estimation technique for finding a practical iteration parameter is presented, which is very effective and practical even for large scale problems. At last, some numerical examples are carried, showing that the theoretical results are valid and convincing.</p>

mathematics, applied
Geometric Interpretation of Several Classical Iterative Methods for Linear System of Equations and Diverse Relaxation Parameter of the Sor Method

Xing-jiang Lu,Lai-i Lei

DOI: https://doi.org/10.1007/s11766-013-3167-x

2013-01-01

Abstract:Two kinds of iterative methods are designed to solve the linear system of equations, we obtain a new interpretation in terms of a geometric concept. Therefore, we have a better insight into the essence of the iterative methods and provide a reference for further study and design. Finally, a new iterative method is designed named as the diverse relaxation parameter of the SOR method which, in particular, demonstrates the geometric characteristics. Many examples prove that the method is quite effective.
A Decentralized Proximal Point-type Method for Saddle Point Problems

Liu Weijie,Mokhtari Aryan,Ozdaglar Asuman,Pattathil Sarath,Shen Zebang,Zheng Nenggan

2019-01-01

Abstract: In this paper, we focus on solving a class of constrained non-convex non-concave saddle point problems in a decentralized manner by a group of nodes in a network. Specifically, we assume that each node has access to a summand of a global objective function and nodes are allowed to exchange information only with their neighboring nodes. We propose a decentralized variant of the proximal point method for solving this problem. We show that when the objective function is $\rho$-weakly convex-weakly concave the iterates converge to approximate stationarity with a rate of $\mathcal{O}(1/\sqrt{T})$ where the approximation error depends linearly on $\sqrt{\rho}$. We further show that when the objective function satisfies the Minty VI condition (which generalizes the convex-concave case) we obtain convergence to stationarity with a rate of $\mathcal{O}(1/\sqrt{T})$. To the best of our knowledge, our proposed method is the first decentralized algorithm with theoretical guarantees for solving a non-convex non-concave decentralized saddle point problem. Our numerical results for training a general adversarial network (GAN) in a decentralized manner match our theoretical guarantees.
Nonlinear Uzawa Methods for Solving Nonsymmetric Saddle Point Problems

Yanhua Cao,Yiqin Lin,Yimin Wei

DOI: https://doi.org/10.1007/bf02896385

2006-01-01

Abstract:In [A new nonlinear Uzawa algorithm for generalized saddle point problems, Appl. Math. Comput., 175(2006), 1432–1454], a nonlinear Uzawa algorithm for solving symmetric saddle point problems iteratively, which was defined by two nonlinear approximate inverses, was considered. In this paper, we extend it to the nonsymmetric case. For the nonsymmetric case, its convergence result is deduced. Moreover, we compare the convergence rates of three nonlinear Uzawa methods and show that our method is more efficient than other nonlinear Uzawa methods in some cases. The results of numerical experiments are presented when we apply them to Navier-Stokes equations discretized by mixed finite elements.
A note on GPIU method for generalized saddle point problems.

Shu-Xin Miao,Yang Cao

DOI: https://doi.org/10.1016/j.amc.2013.12.067

IF: 4.397

2014-01-01

Applied Mathematics and Computation

Abstract:In this note, the generalized parameterized inexact Uzawa method, abbreviated as the GPIU method, for solving the generalized saddle point problems with symmetric positive semi-definite (2,2) block is studied. The convergence of the method is established, which is an extension of the results obtained in a recent paper by Zhou and Zhang (2009) [22].
Convergence Analysis of Generalized Nonlinear Inexact Uzawa Algorithm for Stabilized Saddle Point Problems

Junfeng Lu,Zhenyue Zhang

DOI: https://doi.org/10.1007/s11464-011-0129-6

2011-01-01

Frontiers of Mathematics in China

Abstract:This paper deals with a modified nonlinear inexact Uzawa (MNIU) method for solving the stabilized saddle point problem. The modified Uzawa method is an inexact inner-outer iteration with a variable relaxation parameter and has been discussed in the literature for uniform inner accuracy. This paper focuses on the general case when the accuracy of inner iteration can be variable and the convergence of MNIU with variable inner accuracy, based on a simple energy norm. Sufficient conditions for the convergence of MNIU are proposed. The convergence analysis not only greatly improves the existing convergence results for uniform inner accuracy in the literature, but also extends the convergence to the variable inner accuracy that has not been touched in literature. Numerical experiments are given to show the efficiency of the MNIU algorithm.
A Modified Nonlinear Inexact Uzawa Algorithm with a Variable Relaxation Parameter for the Stabilized Saddle Point Problem.

Junfeng Lu,Zhenyue Zhang

DOI: https://doi.org/10.1137/090756235

IF: 1.908

2010-01-01

SIAM Journal on Matrix Analysis and Applications

Abstract:This paper proposes a modified nonlinear inexact Uzawa (MNIU) algorithm for solving the stabilized saddle point problem, by introducing a variable overrelaxation parameter to speed up convergence. MNIU is an inner-outer iteration method with variable inner accuracy. We give a detailed error analysis for the convergence of MNIU, based upon a newly defined error norm which helps to handle variable inner accuracy for the Uzawa method. We also simply formulate the optimal overrelaxation parameter. Sufficient conditions are given for the convergence of MNIU. We show that MNIU converges in a relatively large range for the variable inner accuracy setting. For constant inner accuracy $\delta$, MNIU is convergent when $\delta<1$, too. Compared with the original nonlinear inexact Uzawa algorithm (NIU) that converges only for constant accuracy and $\delta<1/3$, this is a significant improvement. We also show a practical approach for estimating the optimal overrelaxation parameter for numerical computation. Numerical experiments of MNIU are given and compared with other methods. These results confirm the significant improvements of MNIU.

The Mgaor-Iterative Method for Saddle Point Problems

On the Iterative Algorithm for Large Sparse Saddle Point Problems

An Inexact Uzawa-Type Iterative Method For Solving Saddle Point Problems

A preconditioned iteration method for solving saddle point problems

A special parameterized inexact Uzawa algorithm for symmetric saddle point problem

Multiple Greedy Quasi-Newton Methods for Saddle Point Problems

On Semi-Convergence of the Generalized Shift-Splitting Iteration Method for Singular Nonsymmetric Saddle Point Problems

A Stochastic Proximal Point Algorithm for Saddle-Point Problems

An Iterative Minimization Formulation for Saddle Point Search.

Generalized Optimistic Methods for Convex-Concave Saddle Point Problems

On Preconditioned Generalized Shift-Splitting Iteration Methods for Saddle Point Problems.

The Polynomial Acceleration of GSOR Iterative Algorithms for Large Sparse Saddle Point Problems

A Decentralized Proximal Point-type Method for Non-convex Non-concave Saddle Point Problems

Some Remarks on the Nonlinear Inexact Uzawa Algorithm for Saddle Point Problem

Preconditioned iterative method for nonsymmetric saddle point linear systems

Geometric Interpretation of Several Classical Iterative Methods for Linear System of Equations and Diverse Relaxation Parameter of the Sor Method

A Decentralized Proximal Point-type Method for Saddle Point Problems

Nonlinear Uzawa Methods for Solving Nonsymmetric Saddle Point Problems

A note on GPIU method for generalized saddle point problems.

Convergence Analysis of Generalized Nonlinear Inexact Uzawa Algorithm for Stabilized Saddle Point Problems

A Modified Nonlinear Inexact Uzawa Algorithm with a Variable Relaxation Parameter for the Stabilized Saddle Point Problem.