Abstract:The Newton, Gauss--Newton and Levenberg--Marquardt methods all use the first derivative of a vector function (the Jacobian) to minimise its sum of squares. When the Jacobian matrix is ill-conditioned, the function varies much faster in some directions than others and the space of possible improvement in sum of squares becomes a long narrow ellipsoid in the linear model. This means that even a small amount of nonlinearity in the problem parameters can cause a proposed point far down the long axis of the ellipsoid to fall outside of the actual curved valley of improved values, even though it is quite nearby. This paper presents a differential equation that `follows' these valleys, based on the technique of geodesic acceleration, which itself provides a 2$^\mathrm{nd}$ order improvement to the Levenberg--Marquardt iteration step. Higher derivatives of this equation are computed that allow $n^\mathrm{th}$ order improvements to the optimisation methods to be derived. These higher-order accelerated methods up to 4$^\mathrm{th}$ order are tested numerically and shown to provide substantial reduction of both number of steps and computation time.

On the error estimates of several Newton-like methods1Partial supported by the National Natural Science Foundation of China and by Natural Science Foundation of Zhejiang Province.1

Convergence of Newton'S Method and Inverse Function Theorem in Banach Space

Kantorovich-Type Convergence Criterion for Inexact Newton Methods

Convergence of Newton'S Method for Systems of Equations with Constant Rank Derivatives

Convergence Criterion of Newton's Method for Singular Systems with Constant Rank Derivatives

Kantorovich's type theorems for systems of equations with constant rank derivatives

On the convergence of king-werner iteration method in banach space

Local Convergence of Inexact Methods under the Hölder Condition

A semi-local convergence theorem for a robust revised Newton's method

A Note on the Kantorovich Theorem for Newton Iteration

New high-order convergence iteration methods without employing derivatives for solving nonlinear equations

Newton-like method with some remarks

Convergence Behavior of Gauss-Newton's Method and Extensions of the Smale Point Estimate Theory.

Some Substantial Modifications and Improvements for Derivative-Free Iterative Methods and Derivative-Free Transformation for Multiple Zeros

A robust Kantorovich's theorem on inexact Newton method with relative residual error tolerance

Higher-Order Corrections to Optimisers based on Newton's Method

On Convergence Of A New Secant-Like Method For Solving Nonlinear Equations

Two Conditions Concerning Newton'S Method

The convergence ball of Newton's method and the uniqueness ball of equations under Hölder-type continuous derivatives

A Significant Improvement On Newton'S Iterative Method

A New Semi-Local Convergence Theorem for the Inexact Newton Methods