Abstract:There is a significant expansion in both volume and range of applications along with the concomitant increase in the variety of data sources. These ever-expanding trends have highlighted the necessity for more versatile analysis tools that offer greater opportunities for algorithmic developments and computationally faster operations than the standard flat-view matrix approach. Tensors, or multi-way arrays, provide such an algebraic framework which is naturally suited to data of such large volume, diversity, and veracity. Indeed, the associated tensor decompositions have demonstrated their potential in breaking the Curse of Dimensionality associated with traditional matrix methods, where a necessary exponential increase in data volume leads to adverse or even intractable consequences on computational complexity. A key tool underpinning multi-linear manipulation of tensors and tensor networks is the standard Tensor Contraction Product (TCP). However, depending on the dimensionality of the underlying tensors, the TCP also comes at the price of high computational complexity in tensor manipulation. In this work, we resort to diagrammatic tensor network manipulation to calculate such products in an efficient and computationally tractable manner, by making use of Tensor Train decomposition (TTD). This has rendered the underlying concepts easy to perceive, thereby enhancing intuition of the associated underlying operations, while preserving mathematical rigour. In addition to bypassing the cumbersome mathematical multi-linear expressions, the proposed Tensor Train Contraction Product model is shown to accelerate significantly the underlying computational operations, as it is independent of tensor order and linear in the tensor dimension, as opposed to performing the full computations through the standard approach (exponential in tensor order).

Comb Tensor Networks vs. Matrix Product States: Enhanced Efficiency in High-Dimensional Spaces

Reducing Computational Complexity of Tensor Contractions via Tensor-Train Networks

Cons-training tensor networks

Distributive Pre-Training of Generative Modeling Using Matrix-Product States

Semi-tensor Product-based TensorDecomposition for Neural Network Compression

Comb Convolution for Efficient Convolutional Architecture

Compressing deep neural networks by matrix product operators

Projected Tensor-Tensor Products for Efficient Computation of Optimal Multiway Data Representations

TP: tensor product layer to compress the neural network in deep learning

TensorGPT: Efficient Compression of the Embedding Layer in LLMs based on the Tensor-Train Decomposition

Efficient Contraction of Large Tensor Networks for Weighted Model Counting through Graph Decompositions

Tensor network compressibility of convolutional models

On Stability of Tensor Networks and Canonical Forms

Symmetric Tensor Networks for Generative Modeling and Constrained Combinatorial Optimization

Ultimate tensorization: compressing convolutional and FC layers alike

Compressing multivariate functions with tree tensor networks

The product structure of MPS-under-permutations

Concatenated tensor network states

Parallel Weighted Model Counting with Tensor Networks

Tensorizing Neural Networks

Multi-Tensor Network Representation for High-Order Tensor Completion