Can Newton Method Be Surpassed

Naiyang Deng,Zhaozhi Wang

DOI: https://doi.org/10.1007/bf02884735

1999-01-01

Abstract:A local algorithm is proposed for unconstrained optimization problem. Compared with the traditional Newton method with Choleski factorization, this algorithm has the same quadratic convergence. But its computation cost per iteration in average is less when the dimensionn≥55. The saving is estimated in the theoretical framework.

What problem does this paper attempt to address?

Subspace Truncated-Newton Algorithm for Large-Scale Bound Constrained Optimization

梁昔明,钱积新

DOI: https://doi.org/10.3321/j.issn:1008-9497.2002.05.004

2002-01-01

Abstract:A subspace truncated-Newton algorithm for large-scale bound constrained optimization is proposed. The algorithm uses truncated-Newton method to update the variables with indices outside of the active set while the active variables are updated by the projected gradient method. The global convergence of the method is proved and the numerical comparisons with the performance of subspace limited memory quasi-Newton algorithm on a set of large-scale problems are also made.
A Comparative Study on Four Unconstrained Optimization Algorithms

叶峰,邵之江,梁昔明,钱积新

DOI: https://doi.org/10.3969/j.issn.1000-0984.2004.05.020

2004-01-01

Abstract:This paper investigates the performance of four unconstrained optimization algorithms, including conjugate gradient method, BFGS quasi-Newton method, DFP quasi-Newton method and truncated-Newton method, by three numerical experiments(including a problem with changeable dimension) using Matlab and its toolkit. The results show that truncated-Newton method is the more efficient algorithm for large-scale problems in terms of the execution time.
Gauss-Newton Methods for A Class of Nonsmooth Optimization Problems

C Li,XH Wang

2000-01-01

Progress in Natural Science

Abstract:The local quadratic convergence of the Gauss-Newton method for convex composite optimizations is established for any convex function with a minima set. This work extends Burke and Ferris' results when this minima set is a set of weak sharp minima for the convex function.
Revisiting Sub-sampled Newton Methods

Haishan Ye,Luo,Zhihua Zhang

DOI: https://doi.org/10.48550/arxiv.1608.02875

2016-01-01

Abstract:Many machine learning models depend on solving a large scale optimization problem. Recently, sub-sampled Newton methods have emerged to attract much attention for optimization due to their efficiency at each iteration, rectified a weakness in the ordinary Newton method of suffering a high cost at each iteration while commanding a high convergence rate. In this work we propose two new efficient Newton-type methods, Refined Sub-sampled Newton and Refined Sketch Newton. Our methods exhibit a great advantage over existing sub-sampled Newton methods, especially when Hessian-vector multiplication can be calculated efficiently. Specifically, the proposed methods are shown to converge superlinearly in general case and quadratically under a little stronger assumption. The proposed methods can be generalized to a unifying framework for the convergence proof of several existing sub-sampled Newton methods, revealing new convergence properties. Finally, we empirically evaluate the performance of our methods on several standard datasets and the results show consistent improvement in computational efficiency.
Issues on the use of a modified Bunch and Kaufman decomposition for large scale Newton’s equation

Andrea Caliciotti,Giovanni Fasano,Florian Potra,Massimo Roma

DOI: https://doi.org/10.1007/s10589-020-00225-8

2020-09-18

Computational Optimization and Applications

Abstract:Abstract In this work, we deal with Truncated Newton methods for solving large scale (possibly nonconvex) unconstrained optimization problems. In particular, we consider the use of a modified Bunch and Kaufman factorization for solving the Newton equation, at each (outer) iteration of the method. The Bunch and Kaufman factorization of a tridiagonal matrix is an effective and stable matrix decomposition, which is well exploited in the widely adopted SYMMBK (Bunch and Kaufman in Math Comput 31:163–179, 1977; Chandra in Conjugate gradient methods for partial differential equations, vol 129, 1978; Conn et al. in Trust-region methods. MPS-SIAM series on optimization, Society for Industrial Mathematics, Philadelphia, 2000; HSL, A collection of Fortran codes for large scale scientific computation, http://www.hsl.rl.ac.uk/ ; Marcia in Appl Numer Math 58:449–458, 2008) routine. It can be used to provide conjugate directions, both in the case of $$1\times 1$$ 1 × 1 and $$2\times 2$$ 2 × 2 pivoting steps. The main drawback is that the resulting solution of Newton’s equation might not be gradient–related, in the case the objective function is nonconvex. Here we first focus on some theoretical properties, in order to ensure that at each iteration of the Truncated Newton method, the search direction obtained by using an adapted Bunch and Kaufman factorization is gradient–related. This allows to perform a standard Armijo-type linesearch procedure, using a bounded descent direction. Furthermore, the results of an extended numerical experience using large scale CUTEst problems is reported, showing the reliability and the efficiency of the proposed approach, both on convex and nonconvex problems.

mathematics, applied,operations research & management science
A Multilevel Low-Rank Newton Method with Super-linear Convergence Rate and its Application to Non-convex Problems

Nick Tsipinakis,Panagiotis Tigkas,Panos Parpas

2023-05-15

Abstract:Second-order methods can address the shortcomings of first-order methods for the optimization of large-scale machine learning models. However, second-order methods have significantly higher computational costs associated with the computation of second-order information. Subspace methods that are based on randomization have addressed some of these computational costs as they compute search directions in lower dimensions. Even though super-linear convergence rates have been empirically observed, it has not been possible to rigorously show that these variants of second-order methods can indeed achieve such fast rates. Also, it is not clear whether subspace methods can be applied to non-convex cases. To address these shortcomings, we develop a link between multigrid optimization methods and low-rank Newton methods that enables us to prove the super-linear rates of stochastic low-rank Newton methods rigorously. Our method does not require any computations in the original model dimension. We further propose a truncated version of the method that is capable of solving high-dimensional non-convex problems. Preliminary numerical experiments show that our method has a better escape rate from saddle points compared to accelerated gradient descent and Adam and thus returns lower training errors.

Optimization and Control
A globally convergent proximal Newton-type method in nonsmooth convex optimization

Boris S. Mordukhovich,Xiaoming Yuan,Shangzhi Zeng,Jin Zhang

DOI: https://doi.org/10.1007/s10107-022-01797-5

IF: 2.7

2022-03-22

Mathematical Programming

Abstract:The paper proposes and justifies a new algorithm of the proximal Newton type to solve a broad class of nonsmooth composite convex optimization problems without strong convexity assumptions. Based on advanced notions and techniques of variational analysis, we establish implementable results on the global convergence of the proposed algorithm as well as its local convergence with superlinear and quadratic rates. For certain structured problems, the obtained local convergence conditions do not require the local Lipschitz continuity of the corresponding Hessian mappings that is a crucial assumption used in the literature to ensure a superlinear convergence of other algorithms of the proximal Newton type. The conducted numerical experiments of solving the regularized logistic regression model illustrate the possibility of applying the proposed algorithm to deal with practically important problems.

mathematics, applied,computer science, software engineering,operations research & management science
Subspace Newton method for sparse group optimization problem

Shichen Liao,Congying Han,Tiande Guo,Bonan Li

DOI: https://doi.org/10.1007/s10898-024-01396-y

2024-04-30

Journal of Global Optimization

Abstract:This paper investigates sparse optimization problems characterized by a sparse group structure, where element- and group-level sparsity are jointly taken into account. This particular optimization model has exhibited notable efficacy in tasks such as feature selection, parameter estimation, and the advancement of model interpretability. Central to our study is the scrutiny of the and norm regularization model, which, in comparison to alternative surrogate formulations, presents formidable computational challenges. We embark on our study by conducting the analysis of the optimality conditions of the sparse group optimization problem, leveraging the notion of a -stationary point, whose linkage to local and global minimizer is established. In a subsequent facet of our study, we develop a novel subspace Newton algorithm for sparse group optimization problem and prove its global convergence property as well as local second-order convergence rate. Experimental results reveal the superlative performance of our algorithm in terms of both precision and computational expediency, thereby outperforming several state-of-the-art solvers.

mathematics, applied,operations research & management science
Newton Sketch: A Linear-time Optimization Algorithm with Linear-Quadratic Convergence

Mert Pilanci,Martin J. Wainwright

DOI: https://doi.org/10.48550/arXiv.1505.02250

2015-05-09

Abstract:We propose a randomized second-order method for optimization known as the Newton Sketch: it is based on performing an approximate Newton step using a randomly projected or sub-sampled Hessian. For self-concordant functions, we prove that the algorithm has super-linear convergence with exponentially high probability, with convergence and complexity guarantees that are independent of condition numbers and related problem-dependent quantities. Given a suitable initialization, similar guarantees also hold for strongly convex and smooth objectives without self-concordance. When implemented using randomized projections based on a sub-sampled Hadamard basis, the algorithm typically has substantially lower complexity than Newton's method. We also describe extensions of our methods to programs involving convex constraints that are equipped with self-concordant barriers. We discuss and illustrate applications to linear programs, quadratic programs with convex constraints, logistic regression and other generalized linear models, as well as semidefinite programs.

Optimization and Control,Data Structures and Algorithms,Machine Learning
Approximate Newton Methods and Their Local Convergence.

Haishan Ye,Luo,Zhihua Zhang

2017-01-01

Abstract:Many machine learning models are reformulated as optimization problems. Thus, it is important to solve a large-scale optimization problem in big data applications. Recently, stochastic second order methods have emerged to attract much attention for optimization due to their efficiency at each iteration, rectified a weakness in the ordinary Newton method of suffering a high cost in each iteration while commanding a high convergence rate. However, the convergence properties of these methods are still not well understood. There are also several important gaps between the current convergence theory and the performance in real applications. In this paper, we aim to fill these gaps. We propose a unifying framework to analyze local convergence properties of second order methods. Based on this framework, our theoretical analysis matches the performance in real applications.
A quadratically convergent semismooth Newton method for nonlinear semidefinite programming without generalized Jacobian regularity

Fuxiaoyue Feng,Chao Ding,Xudong Li

2024-11-25

Abstract:We introduce a quadratically convergent semismooth Newton method for nonlinear semidefinite programming that eliminates the need for the generalized Jacobian regularity, a common yet stringent requirement in existing approaches. Our strategy involves identifying a single nonsingular element within the Bouligand generalized Jacobian, thus avoiding the standard requirement for nonsingularity across the entire generalized Jacobian set, which is often too restrictive for practical applications. The theoretical framework is supported by introducing the weak second order condition (W-SOC) and the weak strict Robinson constraint qualification (W-SRCQ). These conditions not only guarantee the existence of a nonsingular element in the generalized Jacobian but also forge a primal-dual connection in linearly constrained convex quadratic programming. The theoretical advancements further lay the foundation for the algorithmic design of a novel semismooth Newton method, which integrates a correction step to address degenerate issues. Particularly, this correction step ensures the local convergence as well as a superlinear/quadratic convergence rate of the proposed method. Preliminary numerical experiments corroborate our theoretical findings and underscore the practical effectiveness of our method.

Optimization and Control
A Conservation Law Method in Optimization

Bin Shi

DOI: https://doi.org/10.48550/arXiv.1708.08035

2017-10-15

Abstract:We propose some algorithms to find local minima in nonconvex optimization and to obtain global minima in some degree from the Newton Second Law without friction. With the key observation of the velocity observable and controllable in the motion, the algorithms simulate the Newton Second Law without friction based on symplectic Euler scheme. From the intuitive analysis of analytical solution, we give a theoretical analysis for the high-speed convergence in the algorithm proposed. Finally, we propose the experiments for strongly convex function, non-strongly convex function and nonconvex function in high-dimension.

Optimization and Control,Artificial Intelligence,Numerical Analysis
Limited-memory common-directions method for large-scale optimization: convergence, parallelization, and distributed optimization

Ching-pei Lee,Po-Wei Wang,Chih-Jen Lin

DOI: https://doi.org/10.1007/s12532-022-00219-z

2022-03-29

Mathematical Programming Computation

Abstract:Abstract In this paper, we present a limited-memory common-directions method for smooth optimization that interpolates between first- and second-order methods. At each iteration, a subspace of a limited dimension size is constructed using first-order information from previous iterations, and an efficient Newton method is deployed to find an approximate minimizer within this subspace. With properly selected subspace of dimension as small as two, the proposed algorithm achieves the optimal convergence rates for first-order methods while remaining a descent method, and it also possesses fast convergence speed on nonconvex problems. Since the major operations of our method are dense matrix-matrix operations, the proposed method can be efficiently parallelized in multicore environments even for sparse problems. By wisely utilizing historical information, our method is also communication-efficient in distributed optimization that uses multiple machines as the Newton steps can be calculated with little communication. Numerical study shows that our method has superior empirical performance on real-world large-scale machine learning problems.
A Matrix-Free Newton Method

Uwe Naumann

2023-05-05

Abstract:A modification of Newton's method for solving systems of $n$ nonlinear equations is presented. The new matrix-free method relies on a given decomposition of the invertible Jacobian of the residual into invertible sparse local Jacobians according to the chain rule of differentiation. It is motivated in the context of local Jacobians with bandwidth $2m+1$ for $m\ll n$. A reduction of the computational cost by $\mathcal{O}(\frac{n}{m})$ can be observed. Supporting run time measurements are presented for the tridiagonal case showing a reduction of the computational cost by $\mathcal{O}(n).$ Generalization yields the combinatorial Matrix-Free Newton Step problem. We prove NP-completeness and we present algorithmic components for building methods for the approximate solution. Inspired by adjoint Algorithmic Differentiation, the new method shares several challenges for the latter including the DAG Reversal problem. Further challenges are due to combinatorial problems in sparse linear algebra such as Bandwidth or Directed Elimination Ordering.

Numerical Analysis,Mathematical Software
A Newton-CG based barrier-augmented Lagrangian method for general nonconvex conic optimization

Chuan He,Heng Huang,Zhaosong Lu

DOI: https://doi.org/10.1007/s10589-024-00603-6

2024-09-02

Computational Optimization and Applications

Abstract:In this paper we consider finding an approximate second-order stationary point (SOSP) of general nonconvex conic optimization that minimizes a twice differentiable function subject to nonlinear equality constraints and also a convex conic constraint. In particular, we propose a Newton-conjugate gradient (Newton-CG) based barrier-augmented Lagrangian method for finding an approximate SOSP of this problem. Under some mild assumptions, we show that our method enjoys a total inner iteration complexity of and an operation complexity of for finding an -SOSP of general nonconvex conic optimization with high probability. Moreover, under a constraint qualification, these complexity bounds are improved to and , respectively. To the best of our knowledge, this is the first study on the complexity of finding an approximate SOSP of general nonconvex conic optimization. Preliminary numerical results are presented to demonstrate superiority of the proposed method over first-order methods in terms of solution quality.

mathematics, applied,operations research & management science
Continuation Newton methods with deflation techniques for global optimization problems

Xin-long Luo,Hang Xiao,Sen Zhang

DOI: https://doi.org/10.1007/s11075-024-01768-1

IF: 2.37

2024-02-23

Numerical Algorithms

Abstract:The global minimum point of an optimization problem is of interest in engineering fields and it is difficult to solve, especially for a nonconvex large-scale optimization problem. In this article, we consider a new memetic algorithm for this problem. That is to say, we use the continuation Newton method with the deflation technique to find multiple stationary points of the objective function and use those found stationary points as the initial seeds of the evolutionary algorithm, other than the random initial seeds of the known evolutionary algorithms. Meanwhile, in order to retain the usability of the derivative-free method and the fast convergence of the gradient-based method, we use the automatic differentiation technique to compute the gradient and replace the Hessian matrix with its finite difference approximation. According to our numerical experiments, this new algorithm works well for unconstrained optimization problems and finds their global minima efficiently, in comparison to the other representative global optimization methods such as the multi-start methods (the built-in subroutine GlobalSearch.m of MATLAB R2021b, GLODS, and VRBBO), the branch-and-bound method (Couenne, a state-of-the-art open-source solver for mixed integer nonlinear programming problems), and the derivative-free algorithms (CMA-ES and MCS).

mathematics, applied
A structured modified Newton approach for solving systems of nonlinear equations arising in interior-point methods for quadratic programming

David Ek,Anders Forsgren

DOI: https://doi.org/10.1007/s10589-023-00486-z

2023-05-03

Abstract:The focus in this work is on interior-point methods for inequality-constrained quadratic programs, and particularly on the system of nonlinear equations to be solved for each value of the barrier parameter. Newton iterations give high quality solutions, but we are interested in modified Newton systems that are computationally less expensive at the expense of lower quality solutions. We propose a structured modified Newton approach where each modified Jacobian is composed of a previous Jacobian, plus one low-rank update matrix per succeeding iteration. Each update matrix is, for a given rank, chosen such that the distance to the Jacobian at the current iterate is minimized, in both 2-norm and Frobenius norm. The approach is structured in the sense that it preserves the nonzero pattern of the Jacobian. The choice of update matrix is supported by results in an ideal theoretical setting. We also produce numerical results with a basic interior-point implementation to investigate the practical performance within and beyond the theoretical framework. In order to improve performance beyond the theoretical framework, we also motivate and construct two heuristics to be added to the method.

Optimization and Control
Subspace Quasi-Newton Method with Gradient Approximation

Taisei Miyaishi,Ryota Nozawa,Pierre-Louis Poirion,Akiko Takeda

2024-06-04

Abstract:In recent years, various subspace algorithms have been developed to handle large-scale optimization problems. Although existing subspace Newton methods require fewer iterations to converge in practice, the matrix operations and full gradient computation are bottlenecks when dealing with large-scale problems. %In this study, We propose a subspace quasi-Newton method that is restricted to a deterministic-subspace together with a gradient approximation based on random matrix theory. Our method does not require full gradients, let alone Hessian matrices. Yet, it achieves the same order of the worst-case iteration complexities in average for convex and nonconvex cases, compared to existing subspace methods. In numerical experiments, we confirm the superiority of our algorithm in terms of computation time.

Optimization and Control
Newton and interior-point methods for (constrained) nonconvex-nonconcave minmax optimization with stability and instability guarantees

Raphael Chinchilla,Guosong Yang,Joao P. Hespanha

2024-02-12

Abstract:We address the problem of finding a local solution to a nonconvex-nonconcave minmax optimization using Newton type methods, including interior-point ones. We modify the Hessian matrix of these methods such that, at each step, the modified Newton update direction can be seen as the solution to a quadratic program that locally approximates the minmax problem. Moreover, we show that by selecting the modification in an appropriate way, the only stable equilibrium points of the algorithm's iterations are local minmax points. As a consequence, the algorithm can only converge towards an equilibrium point if such point is a local minmax, and it will escape if the point is not a local minmax. Using numerical examples, we show that the computation time of our algorithm scales roughly linearly with the number of nonzero elements in the Hessian.

Optimization and Control,Systems and Control

Can Newton Method Be Surpassed

Subspace Truncated-Newton Algorithm for Large-Scale Bound Constrained Optimization

A Comparative Study on Four Unconstrained Optimization Algorithms

Gauss-Newton Methods for A Class of Nonsmooth Optimization Problems

Revisiting Sub-sampled Newton Methods

Issues on the use of a modified Bunch and Kaufman decomposition for large scale Newton’s equation

A Multilevel Low-Rank Newton Method with Super-linear Convergence Rate and its Application to Non-convex Problems

A globally convergent proximal Newton-type method in nonsmooth convex optimization

Subspace Newton method for sparse group optimization problem

Newton Sketch: A Linear-time Optimization Algorithm with Linear-Quadratic Convergence

Approximate Newton Methods and Their Local Convergence.

A quadratically convergent semismooth Newton method for nonlinear semidefinite programming without generalized Jacobian regularity

A Conservation Law Method in Optimization

Limited-memory common-directions method for large-scale optimization: convergence, parallelization, and distributed optimization

A Matrix-Free Newton Method

A Newton-CG based barrier-augmented Lagrangian method for general nonconvex conic optimization

Continuation Newton methods with deflation techniques for global optimization problems

A structured modified Newton approach for solving systems of nonlinear equations arising in interior-point methods for quadratic programming

Subspace Quasi-Newton Method with Gradient Approximation

Newton and interior-point methods for (constrained) nonconvex-nonconcave minmax optimization with stability and instability guarantees