Abstract:In this paper we generalize the technique of deflation to define two new methods to systematically find many local minima of a nonlinear least squares problem. The methods are based on the Gauss-Newton algorithm, and as such do not require the calculation of a Hessian matrix. They also require fewer deflations than for applying the deflated Newton method on the first order optimality conditions, as the latter finds all stationary points, not just local minima. One application of interest covered in this paper is the inverse eigenvalue problem (IEP) associated with the modelling of spectroscopic data of relevance to the physical and chemical sciences. Open source MATLAB code is provided at <a class="link-external link-https" href="https://github.com/AlbanBloorRiley/DeflatedGaussNewton" rel="external noopener nofollow">this https URL</a>.
What problem does this paper attempt to address?
The problem that this paper attempts to solve is: how to systematically find multiple local minima of the nonlinear least - squares problem. Specifically, the authors propose and study two new deflation techniques based on the Gauss - Newton algorithm. These methods can effectively avoid computing the Hessian matrix and only converge to local minima, rather than all stationary points (including local maxima and saddle points). Moreover, these methods require fewer deflation steps compared to the traditional deflated Newton method.
### Background and Motivation
In many nonlinear optimization problems, finding an arbitrary local minimum is often not sufficient to meet the practical application requirements. Sometimes it is necessary to find the global minimum or multiple local minima of interest. For this purpose, this paper introduces deflation techniques, which were originally proposed by Wilkinson in 1963 for solving multiple roots of polynomial equations. Later, Farrell et al. improved it in 2015, making the method more stable.
### Method Overview
The methods proposed in this paper mainly include the following two aspects:
1. **Deflated Gauss - Newton Method**:
- By defining specific deflation operator conditions, it is directly applied to the Gauss - Newton algorithm.
- These new methods do not need to compute the Hessian matrix and will not converge to local maxima or saddle points.
2. **Two New Methods**:
- **"Good" Deflated Gauss - Newton Method**: When the step - direction is conducive to reducing the objective function, use the undeflated Gauss - Newton step - length; otherwise, use the deflated step - length.
- **"Bad" Deflated Gauss - Newton Method**: Solve the nearby optimization problem by modifying the objective function and combine the undeflated method to ensure convergence to the local minimum of the original problem.
### Application Example
One specific application scenario mentioned in the paper is the inverse eigenvalue problem (IEP), which is related to spectral data modeling in physical and chemical sciences. In addition, the authors also provide open - source MATLAB code for readers to conduct experimental verification.
### Formula Summary
For the convenience of understanding, the following are several key formulas:
- The update formula of the Gauss - Newton method:
\[
p_k=\arg\min_p \|r(x_k)+J_r(x_k)p\|^2
\]
- The deflation operator:
\[
\mu(x; y_1,\ldots,y_n)=\left( \sigma+\frac{1}{\|x - y_1\|_2^\theta} \right)\cdots\left( \sigma+\frac{1}{\|x - y_n\|_2^\theta} \right)
\]
- The update formula of the "good" deflated Gauss - Newton method:
\[
x_{k + 1}=x_k+\beta^{-1}p_k\quad\text{if}\quad\langle p_k,\nabla\eta(x_k)\rangle > \epsilon
\]
\[
x_{k + 1}=x_k+\alpha p_k\quad\text{otherwise}
\]
- The update formula of the "bad" deflated Gauss - Newton method:
\[
\hat{p}_k=\beta_1p_k+\beta_2(J_r^T J_r)^{-1}\nabla\eta
\]
where,
\[
\beta_1=\frac{\beta}{\omega},\quad\beta_2 = -\frac{\|P r\|_2^2}{\omega}
\]
In conclusion, this paper aims to provide an effective method to systematically find multiple local minima of the nonlinear least - squares problem through improved deflation techniques.