Abstract:A zeroth-order Hessian estimator aims to recover the Hessian matrix of an objective function at any given point, using minimal finite-difference computations. This paper studies zeroth-order Hessian estimation for low-rank Hessians, from a matrix recovery perspective. Our challenge lies in the fact that traditional matrix recovery techniques are not directly suitable for our scenario. They either demand incoherence assumptions (or its variants), or require an impractical number of finite-difference computations in our setting. To overcome these hurdles, we employ zeroth-order Hessian estimations aligned with proper matrix measurements, and prove new recovery guarantees for these estimators. More specifically, we prove that for a Hessian matrix $H \in \mathbb{R}^{n \times n}$ of rank $r$, $ \mathcal{O}(nr^2 \log^2 n ) $ proper zeroth-order finite-difference computations ensures a highly probable exact recovery of $H$. Compared to existing methods, our method can greatly reduce the number of finite-difference computations, and does not require any incoherence assumptions.
What problem does this paper attempt to address?
This paper aims to solve the problems faced by the zero - order finite - difference method in estimating low - rank Hessian matrices. Specifically, traditional matrix recovery techniques either require unrealistic assumptions (such as the incoherence assumption) or a large number of finite - difference calculations, which are infeasible in high - dimensional cases. Therefore, the paper proposes a new zero - order finite - difference Hessian estimation method. This method can accurately recover low - rank Hessian matrices with high probability through \(O(nr^{2}\log^{2}n)\) appropriate zero - order finite - difference calculations without the need for the incoherence assumption.
### Main contributions of the paper:
1. **New Hessian estimation method**: The paper proposes a new zero - order finite - difference Hessian estimation method, which can effectively utilize the low - rank structure and reduce the number of required finite - difference calculations.
2. **Theoretical guarantee**: It is proved that without the incoherence assumption, the low - rank Hessian matrix can be accurately recovered with high probability through \(O(nr^{2}\log^{2}n)\) zero - order finite - difference calculations.
3. **Overcoming the limitations of existing methods**: Existing matrix recovery methods either require the incoherence assumption or a large number of finite - difference calculations. The method in this paper overcomes these limitations simultaneously.
### Background and motivation of the paper:
- **Importance of Hessian matrices**: In machine learning, optimization, and other mathematical programming problems, Hessian matrices describe the curvature of the objective function and are crucial for understanding the behavior of the objective function.
- **Challenges in zero - order optimization**: In many practical scenarios, although the function values can be accessed, the lack of an analytical form of the objective function makes it difficult to directly calculate the Hessian matrix. Therefore, it is very important to develop zero - order finite - difference Hessian estimators.
- **Advantages of low - rank structures**: On high - dimensional data sets, low - rank structures are ubiquitous in machine learning. Utilizing this structure can improve the efficiency and effectiveness of optimization and learning algorithms.
### Technical details of the paper:
- **Finite - difference scheme**: The paper uses a specific finite - difference scheme to estimate the Hessian matrix, which can be implemented by random vectors \(u\) and \(v\).
- **Matrix measurement operation**: The paper regards the Hessian estimator as a matrix measurement operation and proves the effectiveness of this operation.
- **Theoretical analysis**: Through matrix concentration inequalities and the Cramer - Chernoff method, the paper proves the theoretical guarantee of the new method.
### Conclusion:
The paper proposes a new zero - order finite - difference Hessian estimation method, which can efficiently recover low - rank Hessian matrices without the need for the incoherence assumption. This method not only reduces the required amount of calculation but also provides a theoretical guarantee, providing an effective solution for high - dimensional optimization and machine learning problems.