Abstract:We develop a Levenberg-Marquardt method for minimizing the sum of a smooth nonlinear least-squar es term $f(x) = \tfrac{1}{2} \|F(x)\|_2^2$ and a nonsmooth term $h$. Both $f$ and $h$ may be nonconvex. Steps are computed by minimizing the sum of a regularized linear least-squares model and a model of $h$ using a first-order method such as the proximal gradient method. We establish global convergence to a first-order stationary point of both a trust-region and a regularization variant of the Levenberg-Marquardt method under the assumptions that $F$ and its Jacobian are Lipschitz continuous and $h$ is proper and lower semi-continuous. In the worst case, both methods perform $O(\epsilon^{-2})$ iterations to bring a measure of stationarity below $\epsilon \in (0, 1)$. We report numerical results on three examples: a group-lasso basis-pursuit denoise example, a nonlinear support vector machine, and parameter estimation in neuron firing. For those examples to be implementable, we describe in detail how to evaluate proximal operators for separable $h$ and for the group lasso with trust-region constraint. In all cases, the Levenberg-Marquardt methods perform fewer outer iterations than a proximal-gradient method with adaptive step length and a quasi-Newton trust-region method, neither of which exploit the least-squares structure of the problem. Our results also highlight the need for more sophisticated subproblem solvers than simple first-order methods.

On the Levenberg-Marquardt Methods for Convex Constrained Nonlinear Equations

On the Convergence Rate of the Inexact Levenberg-Marquardt Method

On the Convergence of Levenberg-Marquardt Method for Solving Nonlinear Systems.

Accelerating the Modified Levenberg-Marquardt Method for Nonlinear Equations.

The Modified Levenberg-Marquardt Method for Nonlinear Equations with Cubic Convergence

On the Quadratic Convergence of the Levenberg-Marquardt Method Without Nonsingularity Assumption

Convergence Properties of a Self-adaptive Levenberg-Marquardt Algorithm under Local Error Bound Condition

On the Multi-Point Levenberg–Marquardt Method for Singular Nonlinear Equations

A MODIFIED LEVENBERG-MARQUARDT ALGORITHM FOR SINGULAR SYSTEM OF NONLINEAR EQUATIONS

On the Complexity of a Stochastic Levenberg-Marquardt Method

Inexact Levenberg-Marquardt Method for Nonlinear Equations

On Convergence Rates of Linearized Proximal Algorithms for Convex Composite Optimization with Applications.

A Note on the Levenberg-Marquardt Parameter

The Levenberg–Marquardt method: an overview of modern convergence theories and more

Covengence Analysis of a Generalized Levenberg-Marquardt Method for Possibly Non-Smooth Inverse Problems

A Shamanskii-like Levenberg-Marquardt Method for Nonlinear Equations

Iterative algorithms for equilibrium problems based on proximal-like methods

Convergence analysis of the Gauss–Newton method for convex inclusion and convex-composite optimization problems

A New Adaptive Levenberg–Marquardt Method for Nonlinear Equations and Its Convergence Rate under the Hölderian Local Error Bound Condition

An Efficient Three-Term Conjugate Gradient Method for Nonlinear Monotone Equations with Convex Constraints

A Levenberg-Marquardt Method for Nonsmooth Regularized Least Squares