Abstract:We prove, using the subspace embedding guarantee in a black box way, that one can achieve the spectral norm guarantee for approximate matrix multiplication with a dimensionality-reducing map having $m = O(\tilde{r}/\varepsilon^2)$ rows. Here $\tilde{r}$ is the maximum stable rank, i.e. squared ratio of Frobenius and operator norms, of the two matrices being multiplied. This is a quantitative improvement over previous work of [MZ11, KVZ14], and is also optimal for any oblivious dimensionality-reducing map. Furthermore, due to the black box reliance on the subspace embedding property in our proofs, our theorem can be applied to a much more general class of sketching matrices than what was known before, in addition to achieving better bounds. For example, one can apply our theorem to efficient subspace embeddings such as the Subsampled Randomized Hadamard Transform or sparse subspace embeddings, or even with subspace embedding constructions that may be developed in the future. Our main theorem, via connections with spectral error matrix multiplication shown in prior work, implies quantitative improvements for approximate least squares regression and low rank approximation. Our main result has also already been applied to improve dimensionality reduction guarantees for $k$-means clustering [CEMMP14], and implies new results for nonparametric regression [YPW15]. We also separately point out that the proof of the "BSS" deterministic row-sampling result of [BSS12] can be modified to show that for any matrices $A, B$ of stable rank at most $\tilde{r}$, one can achieve the spectral norm guarantee for approximate matrix multiplication of $A^T B$ by deterministically sampling $O(\tilde{r}/\varepsilon^2)$ rows that can be found in polynomial time. The original result of [BSS12] was for rank instead of stable rank. Our observation leads to a stronger version of a main theorem of [KMST10].

Dominant subspace and low-rank approximations from block Krylov subspaces without a prescribed gap

On the Unreasonable Effectiveness of Single Vector Krylov Methods for Low-Rank Approximation

Convergence Properties Of Krylov Subspace Methods For Singular Linear Systems With Arbitrary Index

Block subspace expansions for eigenvalues and eigenvectors approximation

A Quadratically Convergent Algorithm for Structured Low-Rank Approximation

Limited Memory Block Krylov Subspace Optimization for Computing Dominant Singular Value Decompositions.

Krylov Methods are (nearly) Optimal for Low-Rank Approximation

Learning-Based Low-Rank Approximations

Randomized algorithms for low-rank matrix approximation: Design, analysis, and applications

A general error analysis for randomized low-rank approximation with application to data assimilation

Estimating the numerical range with a Krylov subspace

An Efficient Gauss--Newton Algorithm for Symmetric Low-Rank Product Matrix Approximations

Data-Driven Linear Complexity Low-Rank Approximation of General Kernel Matrices: A Geometric Approach

Krylov methods for low-rank commuting generalized Sylvester equations

Approximation Accuracy of the Krylov Subspaces for Linear Discrete Ill-Posed Problems

Matrix Nearness Problems with Off-Block-Diagonal Rank Constraints

The Krylov Subspaces, Low Rank Approximations and Ritz Values of LSQR for Linear Discrete Ill-Posed Problems: the Multiple Singular Value Case

Adaptive Low-Rank Methods: Problems on Sobolev Spaces

New Subset Selection Algorithms for Low Rank Approximation: Offline and Online

Suboptimal subspace construction for log-determinant approximation

Optimal approximate matrix product in terms of stable rank