Abstract:We present an adaptive trust-region method for unconstrained optimization that allows inexact solutions to the trust-region subproblems. Our method is a simple variant of the classical trust-region method of Sorensen [1]. The method achieves the best possible convergence bound up to an additive log factor, for finding an $\epsilon$-approximate stationary point, i.e., $O( \Delta_f L^{1/2} \epsilon^{-3/2}) + \tilde{O}(1)$ iterations where $L$ is the Lipschitz constant of the Hessian, $\Delta_f$ is the optimality gap, and $\epsilon$ is the termination tolerance for the gradient norm. This improves over existing trust-region methods whose worst-case bound is at least a factor of $L$ worse. We compare our performance with state-of-the-art trust-region (TRU) and cubic regularization (ARC) methods from the GALAHAD library on the CUTEst benchmark set on problems with more than 100 variables. We use fewer function, gradient, and Hessian evaluations than these methods. For instance, our algorithm's median number of gradient evaluations is $23$ compared to $36$ for TRU and $29$ for ARC.
Compared to the conference version of this paper [2], our revised method includes practical enhancements and a refined subproblems termination criterion. These modifications dramatically improved performance, including an order of magnitude reduction in the shifted geometric mean of wall-clock times.
What problem does this paper attempt to address?
The main problem that this paper attempts to solve is to improve the trust - region method in unconstrained optimization problems, especially for finding the worst - case convergence bound of an ϵ - approximate stationary point. Specifically, the authors propose a new adaptive trust - region method, which allows for the inexact solution of the trust - region sub - problems and theoretically achieves the optimal convergence bound of \(O(\Delta f L^{1/2} \epsilon^{-3/2})+\tilde{O}(1)\) iterations, where \(L\) is the Lipschitz constant of the Hessian matrix, \(\Delta f\) is the gap between the initial point and the global minimum, and \(\epsilon\) is the termination tolerance.
### Main Problems and Challenges
1. **Limitations of the Classical Trust - Region Method**:
- Although the classical trust - region method is reliable, in the worst - case scenario, its convergence bound is \(\epsilon^{-2}\), which is worse than that of the cubic regularization Newton's method (CRN), which is \(O(\Delta f L^{1/2} \epsilon^{-3/2})\).
- For example, for some functions with Lipschitz - continuous Hessians, the classical trust - region method may require \(\epsilon^{-2}\) iterations to find an \(\epsilon\)-approximate stationary point.
2. **Complexity and Practicality of Existing Methods**:
- Although some existing methods (such as TRACE, the method proposed by Jiang et al.) can achieve better convergence bounds, they usually introduce additional regularization parameters or change the original trust - region framework, resulting in increased implementation complexity and decreased practical performance.
3. **Balance between Theory and Practice**:
- The goal of the paper is to develop a trust - region method that can achieve the optimal convergence bound theoretically and remain simple and easy to use in practice.
### Solutions
The authors propose a new adaptive trust - region method (CAT, Consistently Adaptive Trust Region Method), with the main improvements including:
- **New Acceptance Criterion**: A modified ratio \(\hat{\rho}_k\) is introduced to evaluate the effectiveness of the search direction. This new ratio improves the predicted reduction by adding a term \(\theta/2 \min\{\|\nabla f(x_k)\|,\|\nabla f(x_k + d_k)\|\} \|d_k\|\), thereby ensuring a significant reduction in the function value during a successful step.
- **Dynamically Adjusting the Trust - Region Radius**: When \(\hat{\rho}_k\geq\beta\), the trust - region radius \(r_{k + 1}\) is updated to \(\max\{\omega_2 \|d_k\|,r_k\}\), instead of the classical \(\omega_2 r_k\). This update method makes the method more flexible and able to perform better in practice.
- **Sub - problem Termination Condition**: A relaxed termination condition is proposed, which allows for the inexact solution of the trust - region sub - problems while ensuring the convergence of the algorithm.
Through these improvements, the new method not only achieves the optimal convergence bound theoretically, but also the experimental results on the CUTEst benchmark test set show that it outperforms existing trust - region and cubic regularization methods in terms of the number of function value, gradient, and Hessian evaluations.
### Summary
This paper solves the problem of the poor convergence bound of the classical trust - region method in the worst - case scenario by proposing a new adaptive trust - region method, and achieves significant improvements in both theory and practice.