Abstract:We consider low-distortion embeddings for subspaces under \emph{entrywise nonlinear transformations}. In particular we seek embeddings that preserve the norm of all vectors in a space $S = \{y: y = f(x)\text{ for }x \in Z\}$, where $Z$ is a $k$-dimensional subspace of $\mathbb{R}^n$ and $f(x)$ is a nonlinear activation function applied entrywise to $x$. When $f$ is the identity, and so $S$ is just a $k$-dimensional subspace, it is known that, with high probability, a random embedding into $O(k/\epsilon^2)$ dimensions preserves the norm of all $y \in S$ up to $(1\pm \epsilon)$ relative error. Such embeddings are known as \emph{subspace embeddings}, and have found widespread use in compressed sensing and approximation algorithms. We give the first low-distortion embeddings for a wide class of nonlinear functions $f$. In particular, we give additive $\epsilon$ error embeddings into $O(\frac{k\log (n/\epsilon)}{\epsilon^2})$ dimensions for a class of nonlinearities that includes the popular Sigmoid SoftPlus, and Gaussian functions. We strengthen this result to give relative error embeddings under some further restrictions, which are satisfied e.g., by the Tanh, SoftSign, Exponential Linear Unit, and many other `soft' step functions and rectifying units. Understanding embeddings for subspaces under nonlinear transformations is a key step towards extending random sketching and compressing sensing techniques for linear problems to nonlinear ones. We discuss example applications of our results to improved bounds for compressed sensing via generative neural networks.

Spline embedding for nonlinear dimensionality reduction

Nonlinear Dimensionality Reduction with Local Spline Embedding

Image Feature Optimization Based on Nonlinear Dimensionality Reduction

Deep Manifold Transformation for Dimension Reduction

Neural Network Emulation of Spatio-temporal Data Using Linear and Nonlinear Dimensionality Reduction

Deep Manifold Transformation for Nonlinear Dimensionality Reduction

Deep Manifold Computing and Visualization Using Elastic Locally Isometric Smoothness

Deep Manifold Computing and Visualization

Riemannian manifold learning for nonlinear dimensionality reduction

Nonlinear Supervised Dimensionality Reduction via Smooth Regular Embeddings

Maximal Linear Embedding for Dimensionality Reduction

Distortion-Free Nonlinear Dimensionality Reduction

Dimensionality Reduction By Using Sparse Reconstruction Embedding

Subspace Embeddings Under Nonlinear Transformations

Linear Dimensionality Reduction Based on Hybrid Structure Preserving Projections

A Spline Kernel-Based Approach for Nonlinear System Identification with Dimensionality Reduction

Deep Nonlinear Sufficient Dimension Reduction

Robust Supervised Spline Embedding

Local Linear Embedding in Dimensionality Reduction Based on Small World Principle

Local Regularized Least-Square Dimensionality Reduction

Local Feature Discriminant Projection