Gauss-Newton Methods for A Class of Nonsmooth Optimization Problems

C Li,XH Wang

2000-01-01

Progress in Natural Science

Abstract:The local quadratic convergence of the Gauss-Newton method for convex composite optimizations is established for any convex function with a minima set. This work extends Burke and Ferris' results when this minima set is a set of weak sharp minima for the convex function.

What problem does this paper attempt to address?

On Convergence of the Gauss-Newton Method for Convex Composite Optimization

Chong Li,Xinghua Wang

DOI: https://doi.org/10.1007/s101070100249

IF: 2.7

2002-01-01

Mathematical Programming

Abstract:The local quadratic convergence of the Gauss-Newton method for convex composite optimization f=h∘F is established for any convex function h with the minima set C, extending Burke and Ferris’ results in the case when C is a set of weak sharp minima for h.
Convergence analysis of the Gauss–Newton method for convex inclusion and convex-composite optimization problems

C. Li,K.F. Ng

DOI: https://doi.org/10.1016/j.jmaa.2011.11.062

IF: 1.417

2012-01-01

Journal of Mathematical Analysis and Applications

Abstract:Using the convex process theory we study the convergence issues of the iterative sequences generated by the Gauss–Newton method for the convex inclusion problem defined by a cone C and a smooth function F (the derivative is denoted by F′). The restriction in our consideration is minimal and, even in the classical case (the initial point x0 is assumed to satisfy the following two conditions: F′ is Lipschitz around x0 and the convex process Tx0, defined by Tx0⋅=F′(x0)⋅−C, is surjective), our results are new in giving sufficient conditions (which are weaker than the known ones and have a remarkable property being affine-invariant) ensuring the convergence of the iterative sequence with initial point x0. The same study is also made for the so-called convex-composite optimization problem (with objective function given as the composite of a convex function with a smooth map).
Majorizing Functions and Convergence of the Gauss-Newton Method for Convex Composite Optimization

Chong Li,K. F. Ng

DOI: https://doi.org/10.1137/06065622x

IF: 2.763

2007-01-01

SIAM Journal on Optimization

Abstract:We introduce a notion of quasi regularity for points with respect to the inclusion F( x). C, where F is a nonlinear Frechet differentiable function from R-v to R-m. When C is the set of minimum points of a convex real- valued function h on Rm and F ' satisfies the L- average Lipschitz condition of Wang, we use the majorizing function technique to establish the semilocal linear/ quadratic convergence of sequences generated by the Gauss - Newton method ( with quasiregular initial points) for the convex composite function h. F. Results are new even when the initial point is regular and F ' is Lipschitz.
Gauss–Newton Method for Convex Composite Optimizations on Riemannian Manifolds

Jin-Hua Wang,Jen-Chih Yao,Chong Li

DOI: https://doi.org/10.1007/s10898-010-9638-1

2011-01-01

Journal of Global Optimization

Abstract:A notion of quasi-regularity is extended for the inclusion problem ${F(p)\in C}$ , where F is a differentiable mapping from a Riemannian manifold M to ${\mathbb R^n}$ . When C is the set of minimum points of a convex real-valued function h on ${\mathbb R^n}$ and DF satisfies the L-average Lipschitz condition, we use the majorizing function technique to establish the semi-local convergence of sequences generated by the Gauss-Newton method (with quasi-regular initial points) for the convex composite function h ◦ F on Riemannian manifold. Two applications are provided: one is for the case of regularities on Riemannian manifolds and the other is for the case when C is a cone and DF(p 0)(·) − C is surjective. In particular, the results obtained in this paper extend the corresponding one in Wang et al. (Taiwanese J Math 13:633–656, 2009).
Convergence and Uniqueness Properties of Gauss-Newton's Method

C Li,WH Zhang,XQ Jin

DOI: https://doi.org/10.1016/s0898-1221(04)90086-7

IF: 3.218

2004-01-01

Computers & Mathematics with Applications

Abstract:The generalized radius and center Lipschitz conditions with L average are introduced to investigate the convergence of Gauss-Newton's method for finding the nonlinear least squares solution of nonlinear equations. The radii of the convergence ball of Gauss-Newton's method and the uniqueness ball of the solution are estimated. Some applications are given.
Convergence Behavior of Gauss-Newton's Method and Extensions of the Smale Point Estimate Theory.

Chong Li,Nuchun Hu,Jinhua Wang

DOI: https://doi.org/10.1016/j.jco.2010.02.001

IF: 1.333

2010-01-01

Journal of Complexity

Abstract:The notions of Lipschitz conditions with L average are introduced to the study of convergence analysis of Gauss–Newton’s method for singular systems of equations. Unified convergence criteria ensuring the convergence of Gauss–Newton’s method for one kind of singular systems of equations with constant rank derivatives are established and unified estimates of radii of convergence balls are also obtained. Applications to some special cases such as the Kantorovich type conditions, γ-conditions and the Smale point estimate theory are provided and some important known results are extended and/or improved.
On the local convergence of the semismooth Newton method for composite optimization

Jinglu Hu,Tonghua Tian,Shaohua Pan,Zaiwen Wen

DOI: https://doi.org/10.48550/arxiv.2211.01127

2022-01-01

Abstract:In this paper, we consider a large class of nonlinear equations derived from first-order type methods for solving composite optimization problems. Traditional approaches to establishing superlinear convergence rates of semismooth Newton-type methods for solving nonlinear equations usually postulate either nonsingularity of the B-Jacobian or smoothness of the equation. We investigate the feasibility of both conditions. For the nonsingularity condition, we present equivalent characterizations in broad generality, and illustrate that they are easy-to-check criteria for some examples. For the smoothness condition, we show that it holds locally for a large class of residual mappings derived from composite optimization problems. Furthermore, we investigate a relaxed version of the smoothness condition - smoothness restricted to certain active manifolds. We present a conceptual algorithm utilizing such structures and prove that it has a superlinear convergence rate.
Extended Newton Methods for Multiobjective Optimization: Majorizing Function Technique and Convergence Analysis

Jinhua Wang,Yaohua Hu,Carisa Kwok Wai Yu,Chong Li,Xiaoqi Yang

DOI: https://doi.org/10.1137/18m1191737

IF: 2.763

2019-01-01

SIAM Journal on Optimization

Abstract:We consider the extended Newton method for approaching a Pareto optimum of a multiobjective optimization problem, establish quadratic convergence criteria, and estimate a radius of convergence ball under the assumption that the Hessians of objective functions satisfy an L-average Lipschitz condition. These convergence theorems significantly improve the corresponding ones in [J. Fliege, L. M. G. Drummond, and B. F. Svaiter, SIAM J. Optim., 20 (2009), pp. 602-626]. As applications of the obtained results, convergence theorems under the classical Lipschitz condition or the gamma-condition are presented for multiobjective optimization, and the global quadratic convergence results of the extended Newton method with Armijo/Goldstein/Wolfe line-search schemes are also provided.
A globally convergent proximal Newton-type method in nonsmooth convex optimization

Boris S. Mordukhovich,Xiaoming Yuan,Shangzhi Zeng,Jin Zhang

DOI: https://doi.org/10.1007/s10107-022-01797-5

IF: 2.7

2022-03-22

Mathematical Programming

Abstract:The paper proposes and justifies a new algorithm of the proximal Newton type to solve a broad class of nonsmooth composite convex optimization problems without strong convexity assumptions. Based on advanced notions and techniques of variational analysis, we establish implementable results on the global convergence of the proposed algorithm as well as its local convergence with superlinear and quadratic rates. For certain structured problems, the obtained local convergence conditions do not require the local Lipschitz continuity of the corresponding Hessian mappings that is a crucial assumption used in the literature to ensure a superlinear convergence of other algorithms of the proximal Newton type. The conducted numerical experiments of solving the regularized logistic regression model illustrate the possibility of applying the proposed algorithm to deal with practically important problems.

mathematics, applied,computer science, software engineering,operations research & management science
A linearly convergent Gauss-Newton subgradient method for ill-conditioned problems

Damek Davis,Tao Jiang

DOI: https://doi.org/10.48550/arXiv.2212.13278

2022-12-27

Abstract:We analyze a preconditioned subgradient method for optimizing composite functions $h \circ c$, where $h$ is a locally Lipschitz function and $c$ is a smooth nonlinear mapping. We prove that when $c$ satisfies a constant rank property and $h$ is semismooth and sharp on the image of $c$, the method converges linearly. In contrast to standard subgradient methods, its oracle complexity is invariant under reparameterizations of $c$.

Optimization and Control
A Local Nearly Linearly Convergent First-Order Method for Nonsmooth Functions with Quadratic Growth

Damek Davis,Liwei Jiang

DOI: https://doi.org/10.1007/s10208-024-09653-y

IF: 3.4388

2024-06-16

Foundations of Computational Mathematics

Abstract:Classical results show that gradient descent converges linearly to minimizers of smooth strongly convex functions. A natural question is whether there exists a locally nearly linearly convergent method for nonsmooth functions with quadratic growth. This work designs such a method for a wide class of nonsmooth and nonconvex locally Lipschitz functions, including max-of-smooth, Shapiro's decomposable class, and generic semialgebraic functions. The algorithm is parameter-free and derives from Goldstein's conceptual subgradient method.

mathematics, applied,computer science, theory & methods
Convergence Analysis of the Gauss–Newton-Type Method for Lipschitz-Like Mappings

M. H. Rashid,S. H. Yu,C. Li,S. Y. Wu

DOI: https://doi.org/10.1007/s10957-012-0206-3

2012-01-01

Abstract:We introduce in the present paper a Gauss–Newton-type method for solving generalized equations defined by sums of differentiable mappings and set-valued mappings in Banach spaces. Semi-local convergence and local convergence of the Gauss–Newton-type method are analyzed.
Convergence of the Gauss-Newton method for convex composite optimization problems under majorant condition on Riemannian manifolds

Qamrul Hasan Ansari,Moin Uddin,Jen-Chih Yao

DOI: https://doi.org/10.1016/j.jco.2023.101788

IF: 1.333

2024-02-01

Journal of Complexity

Abstract:In this paper, we consider convex composite optimization problems on Riemannian manifolds, and discuss the semi-local convergence of the Gauss-Newton method with quasi-regular initial point and under the majorant condition. As special cases, we also discuss the convergence of the sequence generated by the Gauss-Newton method under Lipschitz-type condition, or under γ-condition.

mathematics, applied
Coderivative-Based Newton Methods in Structured Nonconvex and Nonsmooth Optimization

Pham Duy Khanh,Boris S. Mordukhovich,Vo Thanh Phat

2024-03-07

Abstract:This paper proposes and develops new Newton-type methods to solve structured nonconvex and nonsmooth optimization problems with justifying their fast local and global convergence by means of advanced tools of variational analysis and generalized differentiation. The objective functions belong to a broad class of prox-regular functions with specification to constrained optimization of nonconvex structured sums. We also develop a novel line search method, which is an extension of the proximal gradient algorithm while allowing us to globalize the proposed coderivative-based Newton methods by incorporating the machinery of forward-backward envelopes. Further applications and numerical experiments are conducted for the $\ell_0$-$\ell_2$ regularized least-square model appearing in statistics and machine learning.

Optimization and Control
Coderivative-Based Newton Methods with Wolfe Linesearch for Nonsmooth Optimization

Miantao Chao,Boris S. Mordukhovich,Zijian Shi,Jin Zhang

2024-07-03

Abstract:This paper introduces and develops novel coderivative-based Newton methods with Wolfe linesearch conditions to solve various classes of problems in nonsmooth optimization. We first propose a generalized regularized Newton method with Wolfe linesearch (GRNM-W) for unconstrained $C^{1,1}$ minimization problems (which are second-order nonsmooth) and establish global as well as local superlinear convergence of their iterates. To deal with convex composite minimization problems (which are first-order nonsmooth and can be constrained), we combine the proposed GRNM-W with two algorithmic frameworks: the forward-backward envelope and the augmented Lagrangian method resulting in the two new algorithms called CNFB and CNAL, respectively. Finally, we present numerical results to solve Lasso and support vector machine problems appearing in, e.g., machine learning and statistics, which demonstrate the efficiency of the proposed algorithms.

Optimization and Control
On the Local Convergence of a Stochastic Semismooth Newton Method for Nonsmooth Nonconvex Optimization

Milzarek Andre,Xiao Xiantao,Wen Zaiwen,Ulbrich Michael

DOI: https://doi.org/10.1007/s11425-020-1865-1

2022-01-01

Science China Mathematics

Abstract:In this work, we present probabilistic local convergence results for a stochastic semismooth Newton method for a class of stochastic composite optimization problems involving the sum of smooth nonconvex and nonsmooth convex terms in the objective function. We assume that the gradient and Hessian information of the smooth part of the objective function can only be approximated and accessed via calling stochastic firstand second-order oracles. The approach combines stochastic semismooth Newton steps, stochastic proximal gradient steps and a globalization strategy based on growth conditions. We present tail bounds and matrix concentration inequalities for the stochastic oracles that can be utilized to control the approximation errors via appropriately adjusting or increasing the sampling rates. Under standard local assumptions, we prove that the proposed algorithm locally turns into a pure stochastic semismooth Newton method and converges r-linearly or r-superlinearly with high probability.
Convex and Non-convex Optimization Under Generalized Smoothness

Haochuan Li,Jian Qian,Yi Tian,Alexander Rakhlin,Ali Jadbabaie

2023-11-03

Abstract:Classical analysis of convex and non-convex optimization methods often requires the Lipshitzness of the gradient, which limits the analysis to functions bounded by quadratics. Recent work relaxed this requirement to a non-uniform smoothness condition with the Hessian norm bounded by an affine function of the gradient norm, and proved convergence in the non-convex setting via gradient clipping, assuming bounded noise. In this paper, we further generalize this non-uniform smoothness condition and develop a simple, yet powerful analysis technique that bounds the gradients along the trajectory, thereby leading to stronger results for both convex and non-convex optimization problems. In particular, we obtain the classical convergence rates for (stochastic) gradient descent and Nesterov's accelerated gradient method in the convex and/or non-convex setting under this general smoothness condition. The new analysis approach does not require gradient clipping and allows heavy-tailed noise with bounded variance in the stochastic setting.

Optimization and Control,Machine Learning
Developing Lagrangian-based Methods for Nonsmooth Nonconvex Optimization

Nachuan Xiao,Kuangyu Ding,Xiaoyin Hu,Kim-Chuan Toh

2024-04-15

Abstract:In this paper, we consider the minimization of a nonsmooth nonconvex objective function $f(x)$ over a closed convex subset $\mathcal{X}$ of $\mathbb{R}^n$, with additional nonsmooth nonconvex constraints $c(x) = 0$. We develop a unified framework for developing Lagrangian-based methods, which takes a single-step update to the primal variables by some subgradient methods in each iteration. These subgradient methods are ``embedded'' into our framework, in the sense that they are incorporated as black-box updates to the primal variables. We prove that our proposed framework inherits the global convergence guarantees from these embedded subgradient methods under mild conditions. In addition, we show that our framework can be extended to solve constrained optimization problems with expectation constraints. Based on the proposed framework, we show that a wide range of existing stochastic subgradient methods, including the proximal SGD, proximal momentum SGD, and proximal ADAM, can be embedded into Lagrangian-based methods. Preliminary numerical experiments on deep learning tasks illustrate that our proposed framework yields efficient variants of Lagrangian-based methods with convergence guarantees for nonconvex nonsmooth constrained optimization problems.

Optimization and Control,Machine Learning
A projected semismooth Newton method for a class of nonconvex composite programs with strong prox-regularity

Jiang Hu,Kangkang Deng,Jiayuan Wu,Quanzheng Li

2023-03-10

Abstract:This paper aims to develop a Newton-type method to solve a class of nonconvex composite programs. In particular, the nonsmooth part is possibly nonconvex. To tackle the nonconvexity, we develop a notion of strong prox-regularity which is related to the singleton property, Lipschitz continuity, and monotonicity of the associated proximal operator, and we verify it in various classes of functions, including weakly convex functions, indicator functions of proximally smooth sets, and two specific sphere-related nonconvex nonsmooth functions. In this case, the problem class we are concerned with covers smooth optimization problems on manifold and certain composite optimization problems on manifold. For the latter, the proposed algorithm is the first second-order type method. Combining with the semismoothness of the proximal operator, we design a projected semismooth Newton method to find a root of the natural residual induced by the proximal gradient method. Since the corresponding natural residual may not be globally monotone, an extra projection is added on the usual semismooth Newton step and new criteria are proposed for the switching between the projected semismooth Newton step and the proximal step. The global convergence is then established under the strong prox-regularity. Based on the BD regularity condition, we establish local superlinear convergence. Numerical experiments demonstrate the effectiveness of our proposed method compared with state-of-the-art ones.

Optimization and Control

Gauss-Newton Methods for A Class of Nonsmooth Optimization Problems

On Convergence of the Gauss-Newton Method for Convex Composite Optimization

Convergence analysis of the Gauss–Newton method for convex inclusion and convex-composite optimization problems

Majorizing Functions and Convergence of the Gauss-Newton Method for Convex Composite Optimization

Gauss–Newton Method for Convex Composite Optimizations on Riemannian Manifolds

Convergence and Uniqueness Properties of Gauss-Newton's Method

Convergence Behavior of Gauss-Newton's Method and Extensions of the Smale Point Estimate Theory.

On the local convergence of the semismooth Newton method for composite optimization

Extended Newton Methods for Multiobjective Optimization: Majorizing Function Technique and Convergence Analysis

A globally convergent proximal Newton-type method in nonsmooth convex optimization

A linearly convergent Gauss-Newton subgradient method for ill-conditioned problems

A Local Nearly Linearly Convergent First-Order Method for Nonsmooth Functions with Quadratic Growth

Convergence Analysis of the Gauss–Newton-Type Method for Lipschitz-Like Mappings

Convergence of the Gauss-Newton method for convex composite optimization problems under majorant condition on Riemannian manifolds

Coderivative-Based Newton Methods in Structured Nonconvex and Nonsmooth Optimization

Coderivative-Based Newton Methods with Wolfe Linesearch for Nonsmooth Optimization

On the Local Convergence of a Stochastic Semismooth Newton Method for Nonsmooth Nonconvex Optimization

Convex and Non-convex Optimization Under Generalized Smoothness

Developing Lagrangian-based Methods for Nonsmooth Nonconvex Optimization

A projected semismooth Newton method for a class of nonconvex composite programs with strong prox-regularity