Abstract:There is a significant expansion in both volume and range of applications along with the concomitant increase in the variety of data sources. These ever-expanding trends have highlighted the necessity for more versatile analysis tools that offer greater opportunities for algorithmic developments and computationally faster operations than the standard flat-view matrix approach. Tensors, or multi-way arrays, provide such an algebraic framework which is naturally suited to data of such large volume, diversity, and veracity. Indeed, the associated tensor decompositions have demonstrated their potential in breaking the Curse of Dimensionality associated with traditional matrix methods, where a necessary exponential increase in data volume leads to adverse or even intractable consequences on computational complexity. A key tool underpinning multi-linear manipulation of tensors and tensor networks is the standard Tensor Contraction Product (TCP). However, depending on the dimensionality of the underlying tensors, the TCP also comes at the price of high computational complexity in tensor manipulation. In this work, we resort to diagrammatic tensor network manipulation to calculate such products in an efficient and computationally tractable manner, by making use of Tensor Train decomposition (TTD). This has rendered the underlying concepts easy to perceive, thereby enhancing intuition of the associated underlying operations, while preserving mathematical rigour. In addition to bypassing the cumbersome mathematical multi-linear expressions, the proposed Tensor Train Contraction Product model is shown to accelerate significantly the underlying computational operations, as it is independent of tensor order and linear in the tensor dimension, as opposed to performing the full computations through the standard approach (exponential in tensor order).

TensorKrowch: Smooth integration of tensor networks in machine learning

TensorTable: Extending PyTorch for mixed relational and linear algebra pipelines

Learning tensor networks with tensor cross interpolation: new algorithms and libraries

TensorNetwork: A Library for Physics and Machine Learning

KHNNs: hypercomplex neural networks computations via Keras using TensorFlow and PyTorch

Tensor-based framework for training flexible neural networks

Optimal Kernel Orchestration for Tensor Programs with Korch

Kriging Convolutional Networks

Kaolin: A PyTorch Library for Accelerating 3D Deep Learning Research

ThunderKittens: Simple, Fast, and Adorable AI Kernels

TorchKGE: Knowledge Graph Embedding in Python and PyTorch

Tensor decompositions and algorithms, with applications to tensor learning

Convolutions Through the Lens of Tensor Networks

Graph Tensor Networks: An Intuitive Framework for Designing Large-Scale Neural Learning Systems on Multiple Domains

A Unifying Tensor View for Lightweight CNNs

Kaleidoscope: An Efficient, Learnable Representation For All Structured Linear Maps

GTCO: Graph and Tensor Co-Design for Transformer-Based Image Recognition on Tensor Cores

Convolutions and More as Einsum: A Tensor Network Perspective with Advances for Second-Order Methods

A Data-Centric Optimization Framework for Machine Learning

Reducing Computational Complexity of Tensor Contractions via Tensor-Train Networks

PyRCN: A Toolbox for Exploration and Application of Reservoir Computing Networks