Abstract:Deep neural network (DNN) compression has become a hot topic in the research of deep learning since the scale of modern DNNs turns into too huge to implement on practical resource constrained platforms such as embedded devices. Among variant compression methods, tensor decomposition appears to be a relatively simple and efficient strategy owing to its solid mathematical foundations and regular data structure. Generally, tensorizing neural weights into higher-order tensors for better decomposition, and directly mapping efficient tensor structure to neural architecture with nonlinear activation functions, are the two most common ways. However, the considerable accuracy loss is still a fly in the ointment for the tensorizing way especially for convolutional neural networks (CNNs), while the number of studies in the mapping way is comparatively limited and corresponding compression ratio appears to be not considerable. Therefore, in this work, by researching multiple types of tensor decompositions, we realize that tensor train (TT), which has specific and efficient sequenced contractions, is potential to take into account both of tensorizing and mapping ways. Then we propose a novel nonlinear tensor train (NTT) format, which contains extra nonlinear activation functions embedded in sequenced contractions and convolutions on the top of the normal TT decomposition and the proposed TT format connected by convolutions, to compensate the accuracy loss that normal TT cannot give. Further than just shrinking the space complexity of original weight matrices and convolutional kernels, we prove that NTT can afford an efficient inference time as well. Extensive experiments and discussions demonstrate that the compressed DNNs in our NTT format can almost maintain the accuracy at least on MNIST, UCF11 and CIFAR-10 datasets, and the accuracy loss caused by normal TT could be compensated significantly on large-scale datasets such as ImageNet.

Nonnegative Tensor Train Decompositions for Multi-domain Feature Extraction and Clustering.

Tensor Denoising Using Low-Rank Tensor Train Decomposition

Distributed Non-Negative Tensor Train Decomposition

MR-NTD: Manifold Regularization Nonnegative Tucker Decomposition for Tensor Data Dimension Reduction and Representation.

Faster Tensor Train Decomposition for Sparse Data

Multi-Branch Tensor Network Structure for Tensor-Train Discriminant Analysis

Adaptive graph regularized non-negative Tucker decomposition for multiway dimensionality reduction

Graph regularized discriminative nonnegative tucker decomposition for tensor data representation

Research on exponential regularization approach for nonnegative Tucker3 decomposition

Non-negative Tri-factor Tensor Decomposition with Applications

Nonlinear Tensor Train Format for Deep Neural Network Compression

Non-negative Tucker decomposition with graph regularization and smooth constraint for clustering

Low-Rank Tensor Networks for Dimensionality Reduction and Large-Scale Optimization Problems: Perspectives and Challenges PART 1

TENSOR TRAIN RANK MINIMIZATION WITH NONLOCAL SELF-SIMILARITY FOR TENSOR COMPLETION

Efficient Tensor Robust PCA under Hybrid Model of Tucker and Tensor Train

Orthogonal Nonnegative Tucker Decomposition

Deep Non-Negative Tensor Factorization with Multi-Way EMG Data.

HyperNTF: A Hypergraph Regularized Nonnegative Tensor Factorization for Dimensionality Reduction

A Generalized Graph Regularized Non-Negative Tucker Decomposition Framework for Tensor Data Representation

A Two Stage Algorithm for K-Mode Convolutive Nonnegative Tucker Decomposition

Nonnegative Low Rank Tensor Approximation and its Application to Multi-dimensional Images