Abstract:SIAM Journal on Scientific Computing, Volume 44, Issue 4, Page C287-C309, August 2022. There are several factorizations of multidimensional tensors into lower-dimensional components, known as "tensor networks." We consider the popular "tensor-train" (TT) format and ask, How efficiently can we compute a low-rank approximation from a full tensor on current multicore CPUs? Compared to sparse and dense linear algebra, kernel libraries for multilinear algebra are rare and typically not as well optimized. Linear algebra libraries like BLAS and LAPACK may provide the required operations in principle but often at the cost of additional data movements for rearranging memory layouts. Furthermore, these libraries are typically optimized for the compute-bound case (e.g., square matrix operations), whereas low-rank tensor decompositions lead to memory bandwidth limited operations. We propose a "TT singular value decomposition" (TT-SVD) algorithm based on two building blocks: a "Q-less tall-skinny QR" factorization and a fused tall-skinny matrix-matrix multiplication and reshape operation. We analyze the performance of the resulting TT-SVD algorithm using the roofline performance model. In addition, we present performance results for different algorithmic variants for shared-memory as well as distributed-memory architectures. Our experiments show that commonly used TT-SVD implementations suffer severe performance penalties. We conclude that a dedicated library for tensor factorization kernels would benefit the community: Computing a low-rank approximation can be as cheap as reading the data twice from main memory. As a consequence, an implementation that achieves realistic performance will move the limit at which one has to resort to randomized methods that only process part of the data.

Matrices with Low-Rank-Plus-Shift Structure: Partial SVD and Latent Semantic Indexing

Document Clustering Using Locality Preserving Indexing

Orthogonal Locality Preserving Indexing

A Simplified Latent Semantic Indexing Approach for Multi-Linguistic Information Retrieval.

The Singular Value Decomposition, Applications and Beyond.

Accelerated singular value thresholding for matrix completion.

Optimal Estimation of Shared Singular Subspaces across Multiple Noisy Matrices

Limited Memory Block Krylov Subspace Optimization for Computing Dominant Singular Value Decompositions.

Beyond Low Rank + Sparse: Multiscale Low Rank Matrix Decomposition

Understanding and Enhancing the Folding-In Method in Latent Semantic Indexing.

Better than best low-rank approximation with the singular value decomposition

A class of sparse Johnson--Lindenstrauss transforms and analysis of their extreme singular values

Efficient Alternating Least Squares Algorithms for Low Multilinear Rank Approximation of Tensors

Structure and Perturbation Analysis of Truncated SVDs for Column-Partitioned Matrices

Low-Rank Approximations with Sparse Factors I: Basic Algorithms and Error Analysis

Tensor Space Model for Document Analysis

Exploiting the Structure Effectively and Efficiently in Low Rank Matrix Recovery.

Low-Rank Updates of Matrix Square Roots

Algorithm xxx: Faster Randomized SVD with Dynamic Shifts

Notes on Low-rank Matrix Factorization

Performance of the Low-Rank TT-SVD for Large Dense Tensors on Modern MultiCore CPUs