SVDinsTN: A Tensor Network Paradigm for Efficient Structure Search from Regularized Modeling Perspective

Yu-Bang Zheng,Xi-Le Zhao,Junhua Zeng,Chao Li,Qibin Zhao,Heng-Chao Li,Ting-Zhu Huang

2024-04-05

Abstract:Tensor network (TN) representation is a powerful technique for computer vision and machine learning. TN structure search (TN-SS) aims to search for a customized structure to achieve a compact representation, which is a challenging NP-hard problem. Recent "sampling-evaluation"-based methods require sampling an extensive collection of structures and evaluating them one by one, resulting in prohibitively high computational costs. To address this issue, we propose a novel TN paradigm, named SVD-inspired TN decomposition (SVDinsTN), which allows us to efficiently solve the TN-SS problem from a regularized modeling perspective, eliminating the repeated structure evaluations. To be specific, by inserting a diagonal factor for each edge of the fully-connected TN, SVDinsTN allows us to calculate TN cores and diagonal factors simultaneously, with the factor sparsity revealing a compact TN structure. In theory, we prove a convergence guarantee for the proposed method. Experimental results demonstrate that the proposed method achieves approximately 100 to 1000 times acceleration compared to the state-of-the-art TN-SS methods while maintaining a comparable level of representation ability.

Computer Science

What problem does this paper attempt to address?

The paper aims to address the challenge of high computational cost in Tensor Network Structure Search (TN-SS). Specifically: 1. **Problem Background**: TN representation expresses high-order data by connecting small-sized tensors (called TN cores) through specific operations. It has garnered significant attention in various fields such as computer vision and machine learning. The goal of TN structure search is to find an appropriate TN structure to achieve compact data representation, which is an NP-hard problem. Existing methods mostly adopt a "sampling-evaluation" framework, requiring sampling and evaluating a large number of candidate structures one by one, leading to extremely high computational costs. 2. **Research Objective**: To address the above issue, the paper proposes a new TN paradigm—SVD-inspired TN decomposition (SVDinsTN). By inserting diagonal factors into a fully connected topology, it can optimize the TN structure while computing the TN cores, thereby avoiding the repetitive structure evaluation process and significantly reducing computational costs. 3. **Contribution Overview**: - Proposed the SVDinsTN method, which allows simultaneous optimization of the TN structure while computing the TN cores, greatly reducing computational overhead. - Theoretically proved the convergence of the proposed method and established an upper bound on the TN rank, which is used to design an efficient initialization scheme. - Experimental results show that compared to existing state-of-the-art TN-SS methods, SVDinsTN can achieve approximately 100 to 1000 times speedup while maintaining comparable representation capability.

SVDinsTN: A Tensor Network Paradigm for Efficient Structure Search from Regularized Modeling Perspective

Adaptive Learning of Tensor Network Structures

Bayesian tensor network structure search and its application to tensor completion

Alternating Local Enumeration (TnALE): Solving Tensor Network Structure Search with Fewer Evaluations

SVD-based algorithms for fully-connected tensor network decomposition

STN: Scalable Tensorizing Networks via Structure-Aware Training and Adaptive Compression

Tensor Wheel Decomposition and Its Tensor Completion Application

Vision Transformer Slimming: Multi-Dimension Searching in Continuous Optimization Space

TSTC: Enabling Efficient Training Via Structured Sparse Tensor Compilation

Sparse-View Spectral CT Reconstruction Based on Tensor Decomposition and Total Generalized Variation

Multi-Branch Tensor Network Structure for Tensor-Train Discriminant Analysis

TSVT: Token Sparsification Vision Transformer for Robust RGB-D Salient Object Detection

Adaptively Topological Tensor Network for Multi-view Subspace Clustering

Revisiting High-Order Tensor Singular Value Decomposition From Basic Element Perspective

DSVT: Dynamic Sparse Voxel Transformer with Rotated Sets

Performance of the Low-Rank TT-SVD for Large Dense Tensors on Modern MultiCore CPUs

Optimum Subspace Learning And Error Correction For Tensors

TENSOR COMPLETION VIA GLOBAL LOW-TUBAL-RANKNESS AND NONLOCAL SELF-SIMILARITY

A Closed-Form Solution to Tensor Voting: Theory and Applications

Task-Specific Scene Structure Representations

Symmetric Tensor Networks for Generative Modeling and Constrained Combinatorial Optimization