Abstract:An \ell _p oblivious subspace embedding is a distribution over r \times n matrices \Pi such that for any fixed n \times d matrix A , \[ \Pr _{\Pi }[\textrm {for all }x, \ \Vert Ax\Vert _p \le \Vert \Pi Ax\Vert _p \le \kappa \Vert Ax\Vert _p] \ge 9/10, \] where r is the dimension of the embedding, \kappa is the distortion of the embedding, and for an n -dimensional vector y , \Vert y\Vert _p = (\sum _{i=1}^n |y_i|^p)^{1/p} is the \ell _p -norm. Another important property is the sparsity of \Pi , that is, the maximum number of non-zero entries per column, as this determines the running time of computing \Pi A . While for p = 2 there are nearly optimal tradeoffs in terms of the dimension, distortion, and sparsity, for the important case of 1 \le p \lt 2 , much less was known. In this article, we obtain nearly optimal tradeoffs for \ell _1 oblivious subspace embeddings, as well as new tradeoffs for 1 \lt p \lt 2 . Our main results are as follows: (1) We show for every 1 \le p \lt 2 , any oblivious subspace embedding with dimension r has distortion \[ \kappa = \Omega \left(\frac{1}{\left(\frac{1}{d}\right)^{1 / p} \log ^{2 / p}r + \left(\frac{r}{n}\right)^{1 / p - 1 / 2}}\right). \] When r = {\operatorname{poly}}(d) \ll n in applications, this gives a \kappa = \Omega (d^{1/p}\log ^{-2/p} d) lower bound, and shows the oblivious subspace embedding of Sohler and Woodruff (STOC, 2011) for p = 1 is optimal up to {\operatorname{poly}}(\log (d)) factors. (2) We give sparse oblivious subspace embeddings for every 1 \le p \lt 2 . Importantly, for p = 1 , we achieve r = O(d \log d) , \kappa = O(d \log d) and s = O(\log d) non-zero entries per column. The best previous construction with s \le {\operatorname{poly}}(\log d) is due to Woodruff and Zhang (COLT, 2013), giving \kappa = \Omega (d^2 {\operatorname{poly}}(\log d)) or \kappa = \Omega (d^{3/2} \sqrt {\log n} \cdot {\operatorname{poly}}(\log d)) and r \ge d \cdot {\operatorname{poly}}(\log d) ; in contrast our r = O(d \log d) and \kappa = O(d \log d) are optimal up to {\operatorname{poly}}(\log (d)) factors even for dense matrices. We also give (1) \ell _p oblivious subspace embeddings with an expected 1+\varepsilon number of non-zero entries per column for arbitrarily small \varepsilon \gt 0 , and (2) the first oblivious subspace embeddings for 1 \le p \lt 2 with O(1) -distortion and dimension independent of n . Oblivious subspace embeddings are crucial for distributed and streaming environments, as well as entrywise \ell _p low-rank approximation. Our results give improved algorithms for these applications.

Optimal Oblivious Subspace Embeddings with Near-optimal Sparsity

Optimal Embedding Dimension for Sparse Subspace Embeddings

Sparsity-Dimension Trade-Offs for Oblivious Subspace Embeddings

Tight Bounds for ℓp Oblivious Subspace Embeddings

D S ] 6 A pr 2 01 8 Tight Bounds for l p Oblivious Subspace Embeddings

On a Connection Between Fast and Sparse Oblivious Subspace Embeddings

Adaptive and Oblivious Randomized Subspace Methods for High-Dimensional Optimization: Sharp Analysis and Lower Bounds

Low Rank Approximation and Regression in Input Sparsity Time

Subspace Embeddings Under Nonlinear Transformations

Oblivious subspace embeddings for compressed Tucker decompositions

Consistent Estimation for PCA and Sparse Regression with Oblivious Outliers

Tight Bounds for the Subspace Sketch Problem with Applications

Inapproximability of Sparsest Vector in a Real Subspace

Approximate Orthogonal Sparse Embedding for Dimensionality Reduction

Unraveling the Veil of Subspace RIP Through Near-Isometry on Subspaces

Robust Sparse Mean Estimation via Sum of Squares

Robust Sparse Recovery with Sparse Bernoulli matrices via Expanders

Robust Sparse Regression with Non-Isotropic Designs

On Differentially Private Subspace Estimation in a Distribution-Free Setting

Randomised subspace methods for non-convex optimization, with applications to nonlinear least-squares