Abstract:We study the problem of residual error estimation for matrix and vector norms using a linear sketch. Such estimates can be used, for example, to quickly assess how useful a more expensive low-rank approximation computation will be. The matrix case concerns the Frobenius norm and the task is to approximate the $k$-residual $\|A - A_k\|_F$ of the input matrix $A$ within a $(1+\epsilon)$-factor, where $A_k$ is the optimal rank-$k$ approximation. We provide a tight bound of $\Theta(k^2/\epsilon^4)$ on the size of bilinear sketches, which have the form of a matrix product $SAT$. This improves the previous $O(k^2/\epsilon^6)$ upper bound in (Andoni et al. SODA 2013) and gives the first non-trivial lower bound, to the best of our knowledge. In our algorithm, our sketching matrices $S$ and $T$ can both be sparse matrices, allowing for a very fast update time. We demonstrate that this gives a substantial advantage empirically, for roughly the same sketch size and accuracy as in previous work. For the vector case, we consider the $\ell_p$-norm for $p>2$, where the task is to approximate the $k$-residual $\|x - x_k\|_p$ up to a constant factor, where $x_k$ is the optimal $k$-sparse approximation to $x$. Such vector norms are frequently studied in the data stream literature and are useful for finding frequent items or so-called heavy hitters. We establish an upper bound of $O(k^{2/p}n^{1-2/p}\operatorname{poly}(\log n))$ for constant $\epsilon$ on the dimension of a linear sketch for this problem. Our algorithm can be extended to the $\ell_p$ sparse recovery problem with the same sketching dimension, which seems to be the first such bound for $p > 2$. We also show an $\Omega(k^{2/p}n^{1-2/p})$ lower bound for the sparse recovery problem, which is tight up to a $\mathrm{poly}(\log n)$ factor.

On The Variance of Schatten $p$-Norm Estimation with Gaussian Sketching Matrices

Improved bounds for randomized Schatten norm estimation of numerically low-rank matrices

On the Schatten P-Quasi-norm Minimization for Low-Rank Matrix Recovery

Optimal Sketching for Residual Error Estimation for Matrix and Vector Norms

Norm and Trace Estimation with Random Rank-one Vectors

Statistical properties of sketching algorithms

Efficient Bounds and Estimates for Canonical Angles in Randomized Subspace Approximations

Almost sharp covariance and Wishart-type matrix estimation

Tractable and Scalable Schatten Quasi-Norm Approximations for Rank Minimization

Matrices with Gaussian Noise: Optimal Estimates for Singular Subspace Perturbation

XTrace: Making the Most of Every Sample in Stochastic Trace Estimation

Asymptotics of the Sketched Pseudoinverse

Analysis of stochastic probing methods for estimating the trace of functions of sparse symmetric matrices

On randomized estimators of the Hafnian of a nonnegative matrix

Nonparametric Testing under Randomized Sketching

Online estimation of the asymptotic variance for averaged stochastic gradient algorithms

On Sketching Quadratic Forms

Lower bounds for variances of Poisson functionals

On Sparse variational methods and the Kullback-Leibler divergence between stochastic processes