Abstract:Positive semi-definite matrices commonly occur as normal matrices of least squares problems in statistics or as kernel matrices in machine learning and approximation theory. They are typically large and dense. Thus algorithms to solve systems with such a matrix can be very costly. A core idea to reduce computational complexity is to approximate the matrix by one with a low rank. The optimal and well understood choice is based on the eigenvalue decomposition of the matrix. Unfortunately, this is computationally very expensive. Cheaper methods are based on Gaussian elimination but they require pivoting. We show how invariant matrix theory provides explicit error formulas for an averaged error based on volume sampling. The formula leads to ratios of elementary symmetric polynomials on the eigenvalues. We discuss several bounds for the expected norm of the approximation error and include examples where this expected error norm can be computed exactly. References A. Dax. “On extremum properties of orthogonal quotients matrices”. In: Lin. Alg. Appl. 432.5 (2010), pp. 1234–1257. doi: 10.1016/j.laa.2009.10.034. M. Dereziński and M. W. Mahoney. Determinantal Point Processes in Randomized Numerical Linear Algebra. 2020. url: https://arxiv.org/abs/2005.03185. A. Deshpande, L. Rademacher, S. Vempala, and G. Wang. “Matrix approximation and projective clustering via volume sampling”. In: Proceedings of the Seventeenth Annual ACM-SIAM Symposium on Discrete Algorithm. SODA ’06. Miami, Florida: Society for Industrial and Applied Mathematics, 2006, pp. 1117–1126. url: https://dl.acm.org/doi/abs/10.5555/1109557.1109681. S. A. Goreinov, E. E. Tyrtyshnikov, and N. L. Zamarashkin. “A theory of pseudoskeleton approximations”. In: Lin. Alg. Appl. 261.1 (1997), pp. 1–21. doi: 10.1016/S0024-3795(96)00301-1. M. W. Mahoney and P. Drineas. “CUR matrix decompositions for improved data analysis”. In: Proc. Nat. Acad. Sci. 106.3 (Jan. 20, 2009), pp. 697–702. doi: 10.1073/pnas.0803205106. M. Marcus and L. Lopes. “Inequalities for symmetric functions and Hermitian matrices”. In: Can. J. Math. 9 (1957), pp. 305–312. doi: 10.4153/CJM-1957-037-9.

On the (non)existence of best low-rank approximations of generic IxJx2 arrays

Sums of squares, Hankel index and almost real rank

Better than best low-rank approximation with the singular value decomposition

Matrix Nearness Problems with Off-Block-Diagonal Rank Constraints

Effective Lower Bounds on the Matrix Rank and Their Applications

Low rank approximation of positive semi-definite symmetric matrices using Gaussian elimination and volume sampling

Optimal low-rank approximations of posteriors for linear Gaussian inverse problems on Hilbert spaces

Randomized low-rank approximations beyond Gaussian random matrices

The bounds for the best rank-1 approximation ratio of a finite dimensional tensor space

General Low-rank Matrix Optimization: Geometric Analysis and Sharper Bounds

Sparse and Low-Rank Matrix Decompositions

When big data actually are low-rank, or entrywise approximation of certain function-generated matrices

Low-Rank Approximation via Generalized Reweighted Iterative Nuclear and Frobenius Norms

Low-Rank Approximations with Sparse Factors I: Basic Algorithms and Error Analysis

On the Unreasonable Effectiveness of Single Vector Krylov Methods for Low-Rank Approximation

A Quadratically Convergent Algorithm for Structured Low-Rank Approximation

On the geometric interpretation of the nonnegative rank

Subtracting a best rank-1 approximation may increase tensor rank

Learning-Based Low-Rank Approximations

Semialgebraic Geometry of Nonnegative Tensor Rank