Abstract:There is a significant expansion in both volume and range of applications along with the concomitant increase in the variety of data sources. These ever-expanding trends have highlighted the necessity for more versatile analysis tools that offer greater opportunities for algorithmic developments and computationally faster operations than the standard flat-view matrix approach. Tensors, or multi-way arrays, provide such an algebraic framework which is naturally suited to data of such large volume, diversity, and veracity. Indeed, the associated tensor decompositions have demonstrated their potential in breaking the Curse of Dimensionality associated with traditional matrix methods, where a necessary exponential increase in data volume leads to adverse or even intractable consequences on computational complexity. A key tool underpinning multi-linear manipulation of tensors and tensor networks is the standard Tensor Contraction Product (TCP). However, depending on the dimensionality of the underlying tensors, the TCP also comes at the price of high computational complexity in tensor manipulation. In this work, we resort to diagrammatic tensor network manipulation to calculate such products in an efficient and computationally tractable manner, by making use of Tensor Train decomposition (TTD). This has rendered the underlying concepts easy to perceive, thereby enhancing intuition of the associated underlying operations, while preserving mathematical rigour. In addition to bypassing the cumbersome mathematical multi-linear expressions, the proposed Tensor Train Contraction Product model is shown to accelerate significantly the underlying computational operations, as it is independent of tensor order and linear in the tensor dimension, as opposed to performing the full computations through the standard approach (exponential in tensor order).

Loop Series Expansions for Tensor Networks

Tensor Networks contraction and the Belief Propagation algorithm

Belief propagation for general graphical models with loops

How to Train Unstable Looped Tensor Network

Tensor Loop Reduction via the Baikov Representation and an Auxiliary Vector

On Stability of Tensor Networks and Canonical Forms

Tensor Network Computations That Capture Strict Variationality, Volume Law Behavior, and the Efficient Representation of Neural Network States

Minimum Cost Loop Nests for Contraction of a Sparse Tensor with a Tensor Network

Loop Neural Networks for Parameter Sharing

Reducing Computational Complexity of Tensor Contractions via Tensor-Train Networks

Equivariant Tensor Network Potentials

Block belief propagation algorithm for two-dimensional tensor networks

Tensor networks for $p$-spin models

Hyperoptimized Approximate Contraction of Tensor Networks with Arbitrary Geometry

Fast Search of the Optimal Contraction Sequence in Tensor Networks

Nuclear norm regularized loop optimization for tensor network

Advancing the Understanding of Fixed Point Iterations in Deep Neural Networks: A Detailed Analytical Study

A Riemannian Approach to the Lindbladian Dynamics of a Locally Purified Tensor Network

Towards a Polynomial Algorithm for Optimal Contraction Sequence of Tensor Networks from Trees.

Belief Propagation Reloaded: Learning BP-Layers for Labeling Problems

Bypassing the Exponential Dependency: Looped Transformers Efficiently Learn In-context by Multi-step Gradient Descent