Abstract:The performance of optimization methods is often tied to the spectrum of the objective Hessian. Yet, conventional assumptions, such as smoothness, do often not enable us to make finely-grained convergence statements -- particularly not for non-convex problems. Striving for a more intricate characterization of complexity, we introduce a unique concept termed graded non-convexity. This allows to partition the class of non-convex problems into a nested chain of subclasses. Interestingly, many traditional non-convex objectives, including partially convex problems, matrix factorizations, and neural networks, fall within these subclasses. As a second contribution, we propose gradient methods with spectral preconditioning, which employ inexact top eigenvectors of the Hessian to address the ill-conditioning of the problem, contingent on the grade. Our analysis reveals that these new methods provide provably superior convergence rates compared to basic gradient descent on applicable problem classes, particularly when large gaps exist between the top eigenvalues of the Hessian. Our theory is validated by numerical experiments executed on multiple practical machine learning problems.

A new note on spectra of optimal and superoptimal preconditioned matrices

A survey and some extensions of T. Chan’s preconditioner

A class of optimal preconditioners and their applications

New Preconditioners for Nonsymmetric and Indefinite Saddle Point Linear Systems

The Spectral Properties of the Preconditioned Matrix for Nonsymmetric Saddle Point Problems.

A Short Note on Singular Values of Optimal and Superoptimal Preconditioned Matrices

The spectral properties of the preconditioned matrix for nonsymmetric saddle point problems

Optimal preconditioners for nonsymmetric multilevel Toeplitz systems with application to solving non-local evolutionary partial differential equations

A Modified T. Chan's Preconditioner for Toeplitz Systems.

A modified SSOR preconditioning method for augmented systems

A new preconditioner for indefinite and asymmetric matrices

A Generalization of T. Chan’s Preconditioner

Spectral Properties of a Class of Matrix Splitting Preconditioners for Saddle Point Problems

A New Preconditioner for Solving Weighted Toeplitz Least Squares Problems

Spectral analysis for preconditioning of multi-dimensional Riesz fractional diffusion equations

Spectral Preconditioning for Gradient Methods on Graded Non-convex Functions

A Note on Preconditioning for M-Matrix

Spectral Properties of the Preconditioned Ahss Iteration Method for Generalized Saddle Point Problems

A Note on Constraint Preconditioners for Nonsymmetric Saddle Point Problems

A Class of Preconditioners Based on the (I+s(alpha))-Type Preconditioning Matrices for Solving Linear Systems

Stability Properties of Superoptimal Preconditioner from Numerical Range