Abstract:Positive semi-definite matrices commonly occur as normal matrices of least squares problems in statistics or as kernel matrices in machine learning and approximation theory. They are typically large and dense. Thus algorithms to solve systems with such a matrix can be very costly. A core idea to reduce computational complexity is to approximate the matrix by one with a low rank. The optimal and well understood choice is based on the eigenvalue decomposition of the matrix. Unfortunately, this is computationally very expensive. Cheaper methods are based on Gaussian elimination but they require pivoting. We show how invariant matrix theory provides explicit error formulas for an averaged error based on volume sampling. The formula leads to ratios of elementary symmetric polynomials on the eigenvalues. We discuss several bounds for the expected norm of the approximation error and include examples where this expected error norm can be computed exactly. References A. Dax. “On extremum properties of orthogonal quotients matrices”. In: Lin. Alg. Appl. 432.5 (2010), pp. 1234–1257. doi: 10.1016/j.laa.2009.10.034. M. Dereziński and M. W. Mahoney. Determinantal Point Processes in Randomized Numerical Linear Algebra. 2020. url: https://arxiv.org/abs/2005.03185. A. Deshpande, L. Rademacher, S. Vempala, and G. Wang. “Matrix approximation and projective clustering via volume sampling”. In: Proceedings of the Seventeenth Annual ACM-SIAM Symposium on Discrete Algorithm. SODA ’06. Miami, Florida: Society for Industrial and Applied Mathematics, 2006, pp. 1117–1126. url: https://dl.acm.org/doi/abs/10.5555/1109557.1109681. S. A. Goreinov, E. E. Tyrtyshnikov, and N. L. Zamarashkin. “A theory of pseudoskeleton approximations”. In: Lin. Alg. Appl. 261.1 (1997), pp. 1–21. doi: 10.1016/S0024-3795(96)00301-1. M. W. Mahoney and P. Drineas. “CUR matrix decompositions for improved data analysis”. In: Proc. Nat. Acad. Sci. 106.3 (Jan. 20, 2009), pp. 697–702. doi: 10.1073/pnas.0803205106. M. Marcus and L. Lopes. “Inequalities for symmetric functions and Hermitian matrices”. In: Can. J. Math. 9 (1957), pp. 305–312. doi: 10.4153/CJM-1957-037-9.

Fast Eigen Decomposition for Low-Rank Matrix Approximation

An algorithm for low-rank matrix factorization and its applications

Adaptive Algorithms for Generalized Eigenvalue Decomposition with A Nonquadratic Criterion

Large-Dimensional Positive Definite Covariance Estimation for High Frequency Data via Low-rank and Sparse Matrix Decomposition

Low-rank matrix decomposition in L1-norm by dynamic systems

The eigenvalue decomposition of normal matrices by the decomposition of the skew-symmetric part with applications to orthogonal matrices

Efficient Orthogonal Decomposition with Automatic Basis Extraction for Low-Rank Matrix Approximation

A non-convex approach to low-rank and sparse matrix decomposition.

A Nonconvex Approach to Low-Rank and Sparse Matrix Decomposition with Application to Video Surveillance.

Low rank approximation of positive semi-definite symmetric matrices using Gaussian elimination and volume sampling

Fast Spectrum Estimation of Some Kernel Matrices

Fast and Positive Definite Estimation of Large Covariance Matrix for High-Dimensional Data Analysis

RLS-based Adaptive Algorithms for Generalized Eigen-Decomposition

An Iterative Coordinate Descent Algorithm to Compute Sparse Low-Rank Approximations

Matrix Completion and Low-Rank SVD via Fast Alternating Least Squares

Fast and Low Memory Cost Matrix Factorization: Algorithm, Analysis, and Case Study.

GoDec+: Fast and Robust Low-Rank Matrix Decomposition Based on Maximum Correntropy.

Efficient and Scalable Kernel Matrix Approximations using Hierarchical Decomposition

Entrywise error bounds for low-rank approximations of kernel matrices

Learning-Based Low-Rank Approximations

A Separable Surrogate Function Method for Sparse and Low-Rank Matrices Decomposition