Abstract:As the deep neural networks (DNNs) with satisfied expression ability usually require large scale to gain adequate performance, and deploying large-scale DNNs on resource-limited environments is still a challenge, neural network compression becomes a hot topic nowadays. Among the multiple compression methods, tensor decomposition reveals many specific advantages, such as regular data structure comes from linear algebra, convenient approach of training from scratch, ideal compression ratio, etc. Nevertheless, for some compact neural modules such as two-dimensional/three-dimensional (2D/3D) convolutional kernels, traditional tensor decomposition in the way of approximation has encountered intractable obstacles. Fortunately, some works utilize the tensor mapping approach, which just regards the data structure of tensor decomposition as neural layers, to reconstruct the convolutional kernel into a new lightweight module with several thinner convolutional kernels. The only two flies in the ointment are there lack necessary theories of tensor mapping, and there is still no tensor mapping way to compress high-order three-dimensional convolutional neural networks (3DCNNs). In this paper, we first deeply analyse the tensor mapping theory including convergence and precision, which separately establishes the rationality of tensor mapping and its superiority over the traditional tensor approximation, according to the Lottery Ticket Hypothesis. Then we propose an efficient method termed as 3D-KCPNet, to compress 3DCNNs based on the novel Kronecker canonical polyadic (KCP) tensor decomposition. The proposed method can not only compress the 3D convolutional kernels, but also convert a 3D convolution to efficient 1 × 1 × 1 and 2D depthwise convolutions. The experiments on the video recognition datasets VIVA Challenge, UCF11, UCF50, and UCF101 show that the accuracy of 3D-KCPNet can surpass its original baseline model and the corresponding tensor approximation model.

Sparse Kronecker Canonical Polyadic Decomposition for Convolutional Neural Networks Compression

Compression Method for Convolution Kernel with KCP-FGTD

TEC-CNN: Towards Efficient Compressing Convolutional Neural Nets with Low-rank Tensor Decomposition

Convolutional neural networks compression with low rank and sparse tensor decompositions

Holistic CNN Compression Via Low-Rank Decomposition with Knowledge Transfer.

Kronecker CP Decomposition with Fast Multiplication for Compressing RNNs

Stable Low-rank Tensor Decomposition for Compression of Convolutional Neural Network

3D-KCPNet: Efficient 3DCNNs based on tensor mapping theory

Realistic Acceleration of Neural Networks with Fine-Grained Tensor Decomposition

Exploiting Kernel Sparsity and Entropy for Interpretable CNN Compression

Towards Compact CNNs via Collaborative Compression

Low-Rank+Sparse Tensor Compression for Neural Networks

Convolutional Neural Network Compression Based on Low-Rank Decomposition

Compressing by Learning in a Low-Rank and Sparse Decomposition Form.

Sensitivity-Oriented Layer-Wise Acceleration and Compression for Convolutional Neural Network.

Fast CP‐compression Layer: Tensor CP‐decomposition to Compress Layers in Deep Learning

CMD: Controllable Matrix Decomposition with Global Optimization for Deep Neural Network Compression

A Pruning Method Based on the Dissimilarity of Angle among Channels and Filters

Reduced storage direct tensor ring decomposition for convolutional neural networks compression

Speeding-up and compression convolutional neural networks by low-rank decomposition without fine-tuning

Co-Exploring Structured Sparsification and Low-Rank Tensor Decomposition for Compact DNNs