Abstract:We study the problem of residual error estimation for matrix and vector norms using a linear sketch. Such estimates can be used, for example, to quickly assess how useful a more expensive low-rank approximation computation will be. The matrix case concerns the Frobenius norm and the task is to approximate the $k$-residual $\|A - A_k\|_F$ of the input matrix $A$ within a $(1+\epsilon)$-factor, where $A_k$ is the optimal rank-$k$ approximation. We provide a tight bound of $\Theta(k^2/\epsilon^4)$ on the size of bilinear sketches, which have the form of a matrix product $SAT$. This improves the previous $O(k^2/\epsilon^6)$ upper bound in (Andoni et al. SODA 2013) and gives the first non-trivial lower bound, to the best of our knowledge. In our algorithm, our sketching matrices $S$ and $T$ can both be sparse matrices, allowing for a very fast update time. We demonstrate that this gives a substantial advantage empirically, for roughly the same sketch size and accuracy as in previous work. For the vector case, we consider the $\ell_p$-norm for $p>2$, where the task is to approximate the $k$-residual $\|x - x_k\|_p$ up to a constant factor, where $x_k$ is the optimal $k$-sparse approximation to $x$. Such vector norms are frequently studied in the data stream literature and are useful for finding frequent items or so-called heavy hitters. We establish an upper bound of $O(k^{2/p}n^{1-2/p}\operatorname{poly}(\log n))$ for constant $\epsilon$ on the dimension of a linear sketch for this problem. Our algorithm can be extended to the $\ell_p$ sparse recovery problem with the same sketching dimension, which seems to be the first such bound for $p > 2$. We also show an $\Omega(k^{2/p}n^{1-2/p})$ lower bound for the sparse recovery problem, which is tight up to a $\mathrm{poly}(\log n)$ factor.

Sketchy Decisions: Convex Low-Rank Matrix Optimization with Optimal Storage

Large-Scale Convex Minimization with a Low-Rank Constraint

Fast Stochastic Algorithms for Low-rank and Nonsmooth Matrix Problems

Sketching for Convex and Nonconvex Regularized Least Squares with Sharp Guarantees

Learning-Based Low-Rank Approximations

Fast low-rank estimation by projected gradient descent: General statistical and algorithmic guarantees

Sketching low-rank matrices with a shared column space by convex programming

Near Optimal Frequent Directions for Sketching Dense and Sparse Matrices

Fast Convex Quadratic Optimization Solvers with Adaptive Sketching-based Preconditioners

Optimal Sketching for Residual Error Estimation for Matrix and Vector Norms

Iterative Hessian sketch: Fast and accurate solution approximation for constrained least-squares

Newton Sketch: A Linear-time Optimization Algorithm with Linear-Quadratic Convergence

The Global Optimization Geometry of Low-Rank Matrix Optimization

Randomization or Condensation?: Linear-Cost Matrix Sketching Via Cascaded Compression Sampling.

Optimization over bounded-rank matrices through a desingularization enables joint global and local guarantees

Learning the Positions in CountSketch

Randomized sketch descent methods for non-separable linearly constrained optimization

Localized sketching for matrix multiplication and ridge regression

Mixed-Projection Conic Optimization: A New Paradigm for Modeling Rank Constraints

Tensor-structured sketching for constrained least squares

Nonconvex Optimization Meets Low-Rank Matrix Factorization: An Overview