Abstract:This work discusses tensor network embeddings, which are random matrices ($S$) with tensor network structure. These embeddings have been used to perform dimensionality reduction of tensor network structured inputs $x$ and accelerate applications such as tensor decomposition and kernel regression. Existing works have designed embeddings for inputs $x$ with specific structures, such that the computational cost for calculating $Sx$ is efficient. We provide a systematic way to design tensor network embeddings consisting of Gaussian random tensors, such that for inputs with more general tensor network structures, both the sketch size (row size of $S$) and the sketching computational cost are low. We analyze general tensor network embeddings that can be reduced to a sequence of sketching matrices. We provide a sufficient condition to quantify the accuracy of such embeddings and derive sketching asymptotic cost lower bounds using embeddings that satisfy this condition and have a sketch size lower than any input dimension. We then provide an algorithm to efficiently sketch input data using such embeddings. The sketch size of the embedding used in the algorithm has a linear dependence on the number of sketching dimensions of the input. Assuming tensor contractions are performed with classical dense matrix multiplication algorithms, this algorithm achieves asymptotic cost within a factor of $O(\sqrt{m})$ of our cost lower bound, where $m$ is the sketch size. Further, when each tensor in the input has a dimension that needs to be sketched, this algorithm yields the optimal sketching asymptotic cost. We apply our sketching analysis to inexact tensor decomposition optimization algorithms. We provide a sketching algorithm for CP decomposition that is asymptotically faster than existing work in multiple regimes, and show optimality of an existing algorithm for tensor train rounding.

Fast Sampling Based Sketches for Tensors

A New Sampling Technique for Tensors

Tensor-structured sketching for constrained least squares

Efficient Tensor Contraction via Fast Count Sketch

Practical Sketching Algorithms for Low-Rank Tucker Approximation of Large Tensors

Tensor-Based Sketching Method for the Low-Rank Approximation of Data Streams

Sampling Methods for Inner Product Sketching

Sample Efficient Learning of Factored Embeddings of Tensor Fields

Fast Sampling for Linear Inverse Problems of Vectors and Tensors using Multilinear Extensions

Generative modeling via tensor train sketching

Seeing the Forest from the Trees in Two Looks: Matrix Sketching by Cascaded Bilateral Sampling

On Sketching Quadratic Forms

RTSMS: Randomized Tucker with single-mode sketching

An Efficient Two-Sided Sketching Method for Large-Scale Tensor Decomposition Based on Transformed Domains

Low-rank Tensor Train Decomposition Using TensorSketch

Fast Regression with an $\ell_\infty$ Guarantee

Sketching Algorithms for Sparse Dictionary Learning: PTAS and Turnstile Streaming

Cost-efficient Gaussian Tensor Network Embeddings for Tensor-structured Inputs

Combining Monte Carlo and Tensor-network Methods for Partial Differential Equations via Sketching

Robust recovery of low-rank matrices and low-tubal-rank tensors from noisy sketches