Abstract:This paper studied the problem of solving the system of nonlinear equations ${\bf F}({\bf x})={\bf 0}$, where ${\bf F}:{\mathbb R}^{d}\to{\mathbb R}^d$. We propose Gram-Reduced Levenberg--Marquardt method which updates the Gram matrix ${\bf J}(\cdot)^\top{\bf J}(\cdot)$ in every $m$ iterations, where ${\bf J}(\cdot)$ is the Jacobian of ${\bf F}(\cdot)$. Our method has a global convergence guarantee without relying on any step of line-search or solving sub-problems. We prove our method takes at most $\mathcal{O}(m^2+m^{-0.5}\epsilon^{-2.5})$ iterations to find an $\epsilon$-stationary point of $\frac{1}{2}\|{\bf F}(\cdot)\|^2$, which leads to overall computation cost of $\mathcal{O}(d^3\epsilon^{-1}+d^2\epsilon^{-2})$ by taking $m=\Theta(\epsilon^{-1})$. Our results are strictly better than the cost of $\mathcal{O}(d^3\epsilon^{-2})$ for existing Levenberg--Marquardt methods. We also show the proposed method enjoys local superlinear convergence rate under the non-degenerate assumption. We provide experiments on real-world applications in scientific computing and machine learning to validate the efficiency of the proposed methods.

A Note on the Levenberg-Marquardt Parameter

On a New Updating Rule of the Levenberg–Marquardt Parameter

Inexact Levenberg-Marquardt Method for Nonlinear Equations

An Adaptive Multi-step Levenberg–Marquardt Method

A Levenberg-Marquardt Algorithm with Correction for Singular System of Nonlinear Equations

On the Convergence Rate of the Inexact Levenberg-Marquardt Method

A New Adaptive Levenberg–Marquardt Method for Nonlinear Equations and Its Convergence Rate under the Hölderian Local Error Bound Condition

Convergence Properties of a Self-adaptive Levenberg-Marquardt Algorithm under Local Error Bound Condition

On the Levenberg-Marquardt Methods for Convex Constrained Nonlinear Equations

A Shamanskii-like Levenberg-Marquardt Method for Nonlinear Equations

Global Complexity Bound of the Levenberg–Marquardt Method

Accelerating the Modified Levenberg-Marquardt Method for Nonlinear Equations.

The Modified Levenberg-Marquardt Method for Nonlinear Equations with Cubic Convergence

Convergence Rate of the Levenberg-Marquardt Method under Hölderian Local Error Bound

On inertial Levenberg-Marquardt type methods for solving nonlinear ill-posed operator equations

On the Quadratic Convergence of the Levenberg-Marquardt Method Without Nonsingularity Assumption

Convergence analysis of Levenberg-Marquardt method with Singular Scaling for nonzero residue nonlinear least-squares problems

An Enhanced Levenberg--Marquardt Method via Gram Reduction

Riemannian Levenberg-Marquardt Method with Global and Local Convergence Properties

A Generalized Adaptive Levenberg–Marquardt Method for Solving Nonlinear Ill-Posed Problems

The Levenberg–Marquardt method: an overview of modern convergence theories and more