A CLASS OF MODIFIED BROYDEN METHODS FOR SOLVING NONCONVEX MINIMIZATION PROBLEMS

CHEN Zhong,FAN Chen-jun,HUANG Liang

2008-01-01

Journal of Mathematics

Abstract:In this paper,we discuss numerical methods for solving nonconvex minimization problems, and present a class of modified Broyden methods with Wolfe-line search. Furthermore, we prove global convergence and superlinear convergence of the modified Broyden methods.

What problem does this paper attempt to address?

The convergence of conjugate gradient method with nonmonotone line search

Zhen-Jun Shi,Shengquan Wang,Zhiwei Xu

DOI: https://doi.org/10.1016/j.amc.2010.06.047

IF: 4.397

2010-01-01

Applied Mathematics and Computation

Abstract:The conjugate gradient method is a useful and powerful approach for solving large-scale minimization problems. Liu and Storey developed a conjugate gradient method, which has good numerical performance but no global convergence under traditional line searches such as Armijo line search, Wolfe line search, and Goldstein line search. In this paper we propose a new nonmonotone line search for Liu-Storey conjugate gradient method (LS in short). The new nonmonotone line search can guarantee the global convergence of LS method and has a good numerical performance. By estimating the Lipschitz constant of the derivative of objective functions in the new nonmonotone line search, we can find an adequate step size and substantially decrease the number of functional evaluations at each iteration. Numerical results show that the new approach is effective in practical computation.
A Nonmonotone Broyden Method for Unconstrained Optimization

Guanghui Liu,Lixing Han,Lili Jing

DOI: https://doi.org/10.1080/10556780108805815

2001-01-01

Abstract:In this paper, a nonmonotone Broyden method for unconstrained optimization is proposed and its global convergence is analyzed, Numerical results show that the proposed nonmonotone method is competitive with its monotone counterpart. In particular, our numerical experiments show that nonmonotone strategies can sometimes improve the performance of the Broyden method.
Extended Newton Methods for Multiobjective Optimization: Majorizing Function Technique and Convergence Analysis

Jinhua Wang,Yaohua Hu,Carisa Kwok Wai Yu,Chong Li,Xiaoqi Yang

DOI: https://doi.org/10.1137/18m1191737

IF: 2.763

2019-01-01

SIAM Journal on Optimization

Abstract:We consider the extended Newton method for approaching a Pareto optimum of a multiobjective optimization problem, establish quadratic convergence criteria, and estimate a radius of convergence ball under the assumption that the Hessians of objective functions satisfy an L-average Lipschitz condition. These convergence theorems significantly improve the corresponding ones in [J. Fliege, L. M. G. Drummond, and B. F. Svaiter, SIAM J. Optim., 20 (2009), pp. 602-626]. As applications of the obtained results, convergence theorems under the classical Lipschitz condition or the gamma-condition are presented for multiobjective optimization, and the global quadratic convergence results of the extended Newton method with Armijo/Goldstein/Wolfe line-search schemes are also provided.
A DECOMPOSITION METHOD FOR CONVEX MINIMIZATION PROBLEMS AND ITS APPLICATION

Xu Yifan,Wu Fang

DOI: https://doi.org/10.1007/bf02669680

2001-01-01

Acta Mathematicae Applicatae Sinica English Series

Abstract:In this paper, we present a modified decomposition algorithm and its bundle style variant for convex programming problems with separable structure. We prove that these methods are globally and linearly convergent and discuss the application of the bundle variant in parallel computations.
A Study of the Dennis-Wolkowicz Method on Convex Functions

Guanghui Liu,Lixing Han

DOI: https://doi.org/10.1023/A:1011212022308

2001-01-01

Computational Optimization and Applications

Abstract:In this paper, we analyze the global convergence of the least-change secant method proposed by Dennis and Wolkowicz, when applied to convex objective functions. One of the most distinguished features of this method is that the Dennis-Wolkowicz update doesn't necessarily belong to the Broyden convex family and can be close to the DFP update, but it still has the self-correcting property. We prove that, for convex objective functions, this method with the commonly used Wolfe line search is globally convergent. We also provide some numerical results.
Gauss-Newton Methods for A Class of Nonsmooth Optimization Problems

C Li,XH Wang

2000-01-01

Progress in Natural Science

Abstract:The local quadratic convergence of the Gauss-Newton method for convex composite optimizations is established for any convex function with a minima set. This work extends Burke and Ferris' results when this minima set is a set of weak sharp minima for the convex function.
Global Convergence of BFGS and PRP Methods under a Modified Weak Wolfe–Powell Line Search

Gonglin Yuan,Zengxin Wei,Xiwen Lu

DOI: https://doi.org/10.1016/j.apm.2017.02.008

IF: 5.336

2017-01-01

Applied Mathematical Modelling

Abstract:The BFGS method is one of the most effective quasi-Newton algorithms for optimization problems. However, its global convergence for general functions is still open. In this paper, under a new line search technique, this problem is solved, and it is shown that other methods in the Broyden class also possess this property. Moreover, the global convergence of the PRP method is established in the case of this new line search. Numerical results are reported to show that the new line search technique is competitive to that of the normal line search. (C) 2017 Elsevier Inc. All rights reserved.
The global convergence of the BFGS method with a modified WWP line search for nonconvex functions

Gonglin Yuan,Pengyuan Li,Junyu Lu

DOI: https://doi.org/10.1007/s11075-022-01265-3

IF: 2.37

2022-04-06

Numerical Algorithms

Abstract:The BFGS method, which has great numerical stability, is one of the quasi-Newton line search methods. However, the global convergence of the BFGS method with a Wolfe line search is still an open problem for general functions. Recently, Yuan et al. (Appl. Math. Mode. 47:811–825, 2017) presented a modified weak Wolfe-Powell (WWP) line search that globally converges for general functions; however, they only partially solved this open problem. In this paper, a further modified WWP line search is proposed, and the global convergence of the BFGS method is established under suitable conditions for general functions. The numerical results show that the new line search produces more interesting results than the normal line search. Moreover, a fact engineering problem is studied with the Muskingum model to show the performance of the presented algorithm.

mathematics, applied
A Modified HS Conjugate Gradient Method and Its Global Convergence

ZHANG Yuan-yuan,ZHANG Jun-rong,XIE Bing-lei

DOI: https://doi.org/10.3969/j.issn.1000-5471.2012.08.011

2012-01-01

Abstract:A modification has been made to HS conjugate gradient method;the sufficient descent property of search direction is guaranteed,which has also been established under the condition of inexact line search condition and assumption that the objective function is nonconvex;under mild conditions,the global convergence property has been proved with the strong Wolfe line search;the numerical testing shows that the numerical effect of algorithm is well.
Explicit Superlinear Convergence of Broyden's Method in Nonlinear Equations

Dachao Lin,Haishan Ye,Zhihua Zhang

2021-01-01

Abstract:In this paper, we follow the recent works about the explicit superlinear convergence rate of quasi-Newton methods. We focus on classical Broyden's methods for solving nonlinear equations and establish explicit (local) superlinear convergence if the initial parameter and approximate Jacobian is close enough to the solution. Our results show two natural trade-offs. The first one is between the superlinear convergence rate and the radius of the neighborhood at initialization. The second one is the balance of the initial distance with the solution and its Jacobian. Moreover, our analysis covers two original Broyden's methods: Broyden's good and bad methods. We discover the difference between them in the scope of local convergence region and the condition number dependence.
A two-step Broyden-like method for nonlinear equations

Zhou, Jinchuan

DOI: https://doi.org/10.1007/s11075-024-01827-7

IF: 2.37

2024-04-26

Numerical Algorithms

Abstract:In this paper, based on a nonmonotone derivative-free line search, we propose a two-step Broyden-like method (denoted by TS-BLM) for solving the nonlinear equations. TS-BLM computes one quasi-Newton step at the beginning iterations. When the iterations are close to the solution set of the nonlinear equations, an additional approximate quasi-Newton step is calculated by solving a linear system formed by using the previous Broyden-like matrix. We prove that TS-BLM converges globally under appropriate conditions. Moreover, we analyze the convergence rate of TS-BLM depending on the approximation of the Broyden-like matrix to the Jacobian. Some numerical results are reported to show the superior numerical performances of TS-BLM compared with the traditional BLM.

mathematics, applied
Linearized Proximal Algorithms with Adaptive Stepsizes for Convex Composite Optimization with Applications

Hu Yaohua,Li Chong,Wang Jinhua,Yang Xiaoqi,Zhu Linglingzhi

DOI: https://doi.org/10.1007/s00245-022-09957-x

2023-01-01

Abstract:We propose an inexact linearized proximal algorithm with an adaptive stepsize, together with its globalized version based on the backtracking line-search, to solve a convex composite optimization problem. Under the assumptions of local weak sharp minima of order p (p≥ 1) for the outer convex function and a quasi-regularity condition for the inclusion problem associated to the inner function, we establish superlinear/quadratic convergence results for proposed algorithms. Compared to the linearized proximal algorithms with a constant stepsize proposed in Hu et al. (SIAM J Optim 26(2):1207–1235, 2016), our algorithms own broader applications and higher convergence rates, and the idea of analysis used in the present paper deviates significantly from that of Hu et al. (2016). Numerical applications to the nonnegative inverse eigenvalue problem and the wireless sensor network localization problem indicate that the proposed algorithms are more efficient and robust, and outperform the algorithms in Hu et al. (2016) and some popular algorithms for relevant problems.
Block Broyden's Methods for Solving Nonlinear Equations.

Chengchang Liu,Cheng Chen,Luo,John C.S. Lui

2023-01-01

Abstract:This paper studies quasi-Newton methods for solving nonlinear equations. We propose block variants of both good and bad Broyden's methods, which enjoy explicit local superlinear convergence rates. Our block good Broyden's method has faster condition-number-free convergence rate than existing Broyden's methods because it takes the advantage of multiple rank modification on the Jacobian estimator. On the other hand, our block bad Broyden's method directly estimates the inverse of the Jacobian provably, which reduces the computational cost of the iteration. Our theoretical results provide some new insights on why good Broyden's method outperforms bad Broyden's method in most of the cases. The empirical results also demonstrate the superiority of our methods and validate our theoretical analysis.
A New Three-Term Conjugate Gradient Algorithm with Modified Gradient-Differences for Solving Unconstrained Optimization Problems

Jie Guo,Zhong Wan

DOI: https://doi.org/10.3934/math.2023128

2023-01-01

AIMS Mathematics

Abstract:Unconstrained optimization problems often arise from mining of big data and scientific computing. On the basis of a modified gradient-difference, this article aims to present a new three-term conjugate gradient algorithm to efficiently solve unconstrained optimization problems. Compared with the existing nonlinear conjugate gradient algorithms, the search directions in this algorithm are always sufficiently descent independent of any line search, as well as having conjugacy property. Using the standard Wolfe line search, global and local convergence of the proposed algorithm is proved under mild assumptions. Implementing the developed algorithm to solve 750 benchmark test problems available in the literature, it is shown that the numerical performance of this algorithm is remarkable, especially in comparison with that of the other similar efficient algorithms.
A Generalized Method for Binary Optimization: Convergence Analysis and Applications

Huan Xiong,Mengyang Yu,Li Liu,Fan Zhu,Jie Qin,Fumin Shen,Ling Shao

DOI: https://doi.org/10.1109/tpami.2021.3070753

IF: 23.6

2021-01-01

IEEE Transactions on Pattern Analysis and Machine Intelligence

Abstract:Binary optimization problems (BOPs) arise naturally in many fields, such as information retrieval, computer vision, and machine learning. Most existing binary optimization methods either use continuous relaxation which can cause large quantization errors, or incorporate a highly specific algorithm that can only be used for particular loss functions. To overcome these difficulties, we propose a novel generalized optimization method, named Alternating Binary Matrix Optimization (ABMO), for solving BOPs. ABMO can handle BOPs with/without orthogonality or linear constraints for a large class of loss functions. ABMO involves rewriting the binary, orthogonality and linear constraints for BOPs as an intersection of two closed sets, then iteratively dividing the original problems into several small optimization problems that can be solved as closed forms. To provide a strict theoretical convergence analysis, we add a sufficiently small perturbation and translate the original problem to an approximated problem whose feasible set is continuous. We not only provide rigorous mathematical proof for the convergence to a stationary and feasible point, but also derive the convergence rate of the proposed algorithm. The promising results obtained from four binary optimization tasks validate the superiority and the generality of ABMO compared with the state-of-the-art methods.

computer science, artificial intelligence,engineering, electrical & electronic
An improved Dai‐Liao‐style hybrid conjugate gradient‐based method for solving unconstrained nonconvex optimization and extension to constrained nonlinear monotone equations

Zihang Yuan,Hu Shao,Xiaping Zeng,Pengjie Liu,Xianglin Rong,Jianhao Zhou

DOI: https://doi.org/10.1002/mma.10396

2024-08-21

Mathematical Methods in the Applied Sciences

Abstract:In this work, for unconstrained optimization, we introduce an improved Dai‐Liao‐style hybrid conjugate gradient method based on the hybridization‐based self‐adaptive technique, and the search direction generated fulfills the sufficient descent and trust region properties regardless of any line search. The global convergence is established under standard Wolfe line search and common assumptions. Then, combining the hyperplane projection technique and a new self‐adaptive line search, we extend the proposed conjugate gradient method and obtain an improved Dai‐Liao‐style hybrid conjugate gradient projection method to solve constrained nonlinear monotone equations. Under mild conditions, we obtain its global convergence without Lipschitz continuity. In addition, the convergence rates for the two proposed methods are analyzed, respectively. Finally, numerical experiments are conducted to demonstrate the effectiveness of the proposed methods.

mathematics, applied
ON THE CONVERGENCE OF A MODIFIED CHEBYSHEV-LIKE'S METHOD FOR SOLVING NONLINEAR EQUATIONS

Lin Zheng,Ke Zhang,Liang Chen

DOI: https://doi.org/10.11650/tjm.19.2015.3856

2015-01-01

Taiwanese Journal of Mathematics

Abstract:In this paper, we introduce a modified Chebyshev-like's method with order four and study the semilocal convergence of the method by using majorizing functions for solving nonlinear equations in Banach spaces. We prove an existence-uniqueness theorem and give a priori error bounds which demonstrates the R-order of the method. Moveover, the local convergence of this method is also analyzed. Finally, numerical application on nonlinear integral equations is given to show our approach.
Global convergence of a BFGS-type algorithm for nonconvex multiobjective optimization problems

L. F. Prudente,D. R. Souza

DOI: https://doi.org/10.1007/s10589-024-00571-x

2024-04-11

Abstract:We propose a modified BFGS algorithm for multiobjective optimization problems with global convergence, even in the absence of convexity assumptions on the objective functions. Furthermore, we establish the superlinear convergence of the method under usual conditions. Our approach employs Wolfe step sizes and ensures that the Hessian approximations are updated and corrected at each iteration to address the lack of convexity assumption. Numerical results shows that the introduced modifications preserve the practical efficiency of the BFGS method.

Optimization and Control
A Modified Gauss-Newton-based BFGS Method for Symmetric Nonlinear Equations

YUAN Gong-lin,WEI Zeng-xin,LU Xi-wen

DOI: https://doi.org/10.3969/j.issn.1005-9164.2006.04.014

2006-01-01

Abstract:In this paper,a modified Gauss-Newton-based BFGS method based on the technique of Li and Fukushima [10] is proposed.The given method possesses the global and superlinear convergence under mild conditions.The presented method is better than the normal method for the given problem.
A Branch-and-Bound Based Method for Solving Monotone Optimization Problems

X. L. Sun,J. L. Li

DOI: https://doi.org/10.1007/s10898-005-3843-3

2006-01-01

Journal of Global Optimization

Abstract:Monotone optimization problems are an important class of global optimization problems with various applications. In this paper, we propose a new exact method for monotone optimization problems. The method is of branch-and-bound framework that combines three basic strategies: partition, convexification and local search. The partition scheme is used to construct a union of subboxes that covers the boundary of the feasible region. The convexification outer approximation is then applied to each subbox to obtain an upper bound of the objective function on the subbox. The performance of the method can be further improved by incorporating the method with local search procedure. Illustrative examples describe how the method works. Computational results for small randomly generated problems are reported.

A CLASS OF MODIFIED BROYDEN METHODS FOR SOLVING NONCONVEX MINIMIZATION PROBLEMS

The convergence of conjugate gradient method with nonmonotone line search

A Nonmonotone Broyden Method for Unconstrained Optimization

Extended Newton Methods for Multiobjective Optimization: Majorizing Function Technique and Convergence Analysis

A DECOMPOSITION METHOD FOR CONVEX MINIMIZATION PROBLEMS AND ITS APPLICATION

A Study of the Dennis-Wolkowicz Method on Convex Functions

Gauss-Newton Methods for A Class of Nonsmooth Optimization Problems

Global Convergence of BFGS and PRP Methods under a Modified Weak Wolfe–Powell Line Search

The global convergence of the BFGS method with a modified WWP line search for nonconvex functions

A Modified HS Conjugate Gradient Method and Its Global Convergence

Explicit Superlinear Convergence of Broyden's Method in Nonlinear Equations

A two-step Broyden-like method for nonlinear equations

Linearized Proximal Algorithms with Adaptive Stepsizes for Convex Composite Optimization with Applications

Block Broyden's Methods for Solving Nonlinear Equations.

A New Three-Term Conjugate Gradient Algorithm with Modified Gradient-Differences for Solving Unconstrained Optimization Problems

A Generalized Method for Binary Optimization: Convergence Analysis and Applications

An improved Dai‐Liao‐style hybrid conjugate gradient‐based method for solving unconstrained nonconvex optimization and extension to constrained nonlinear monotone equations

ON THE CONVERGENCE OF A MODIFIED CHEBYSHEV-LIKE'S METHOD FOR SOLVING NONLINEAR EQUATIONS

Global convergence of a BFGS-type algorithm for nonconvex multiobjective optimization problems

A Modified Gauss-Newton-based BFGS Method for Symmetric Nonlinear Equations

A Branch-and-Bound Based Method for Solving Monotone Optimization Problems