Abstract:SIAM Journal on Scientific Computing, Volume 44, Issue 4, Page C287-C309, August 2022. There are several factorizations of multidimensional tensors into lower-dimensional components, known as "tensor networks." We consider the popular "tensor-train" (TT) format and ask, How efficiently can we compute a low-rank approximation from a full tensor on current multicore CPUs? Compared to sparse and dense linear algebra, kernel libraries for multilinear algebra are rare and typically not as well optimized. Linear algebra libraries like BLAS and LAPACK may provide the required operations in principle but often at the cost of additional data movements for rearranging memory layouts. Furthermore, these libraries are typically optimized for the compute-bound case (e.g., square matrix operations), whereas low-rank tensor decompositions lead to memory bandwidth limited operations. We propose a "TT singular value decomposition" (TT-SVD) algorithm based on two building blocks: a "Q-less tall-skinny QR" factorization and a fused tall-skinny matrix-matrix multiplication and reshape operation. We analyze the performance of the resulting TT-SVD algorithm using the roofline performance model. In addition, we present performance results for different algorithmic variants for shared-memory as well as distributed-memory architectures. Our experiments show that commonly used TT-SVD implementations suffer severe performance penalties. We conclude that a dedicated library for tensor factorization kernels would benefit the community: Computing a low-rank approximation can be as cheap as reading the data twice from main memory. As a consequence, an implementation that achieves realistic performance will move the limit at which one has to resort to randomized methods that only process part of the data.

Fast Randomized Tensor Singular Value Thresholding for Low‐rank Tensor Optimization

Randomized Algorithms for the Low Multilinear Rank Approximations of Tensors

A Fast Implementation of Singular Value Thresholding Algorithm using Recycling Rank Revealing Randomized Singular Value Decomposition

Nonconvex Robust High-Order Tensor Completion Using Randomized Low-Rank Approximation

Accelerated singular value thresholding for matrix completion.

Low-Tubal-Rank Tensor Recovery via Factorized Gradient Descent

Fast and Accurate Randomized Algorithms for Low-rank Tensor Decompositions

Randomized algorithms for computing the tensor train approximation and their applications

Efficient Algorithms for Low Tubal Rank Tensor Approximation with Applications to Image Compression, Super-Resolution and Deep Learning

An Efficient Randomized Fixed-Precision Algorithm for Tensor Singular Value Decomposition

On The Tensor Svd And The Optimal Low Rank Orthogonal Approximation Of Tensors

A Novel Tensor Factorization-Based Method with Robustness to Inaccurate Rank Estimation

A Randomized Algorithm for Tensor Singular Value Decomposition using an Arbitrary Number of Passes

Efficient Alternating Least Squares Algorithms for Low Multilinear Rank Approximation of Tensors

The Computation of Low Multilinear Rank Approximations of Tensors Via Power Scheme and Random Projection

Performance of the Low-Rank TT-SVD for Large Dense Tensors on Modern MultiCore CPUs

Very Large-Scale Singular Value Decomposition Using Tensor Train Networks

T-SVD Based Non-convex Tensor Completion and Robust Principal Component Analysis.

Low-rank optimization on Tucker tensor varieties

Fast randomized algorithms for computing the generalized tensor SVD based on the tubal product