Abstract:Convolutional neural networks (CNNs) possess strong expressive capabilities, but with the increasing size and cost of modern CNNs, deploying large-scale CNNs in resource-limited environments remains a challenge, making neural network compression a hot topic in contemporary research. Among various compression methods, tensor decomposition shows numerous unique advantages, such as regular data structure, solid mathematical foundation, convenience in training from scratch, and ideal compression rates. However, most existing tensor decompositions do not perform well in compressing CNNs, struggling to balance time and space complexity, and failing to accelerate CNNs in reality. To address these issues. In this paper, we propose a fine-grained KCP tensor decomposition for CNN convolution kernel compression with considerable compression rate and acceleration capability. First, convolution kernels are compressed using 2nd-order KCP decomposition. The decomposed factor tensors are then converted into a four-layer lightweight convolution kernel structure. This structure conducts continuous contraction computations with the input, transforming the convolution computations into a highly efficient lightweight layered matrix multiplication structure. Secondly, we conduct a comprehensive comparison with other methods and analyze the superiority of our method. Comparisons of computational complexity with other typical tensor compression methods, along with simulation experiments on CIFAR-10 and ImageNet datasets, demonstrate that our method can achieve more efficient comprehensive results. These include acceptable accuracy, significantly reduced model size, and real-world acceleration of network inference times. Finally, we further analyze how different combinations of KT rank and CP rank impact the compression effectiveness and expressive capacity of CNNs.

Stable Low-rank Tensor Decomposition for Compression of Convolutional Neural Network

Convolutional Neural Network Compression Based on Low-Rank Decomposition

Low-rank Tensor Decomposition for Compression of Convolutional Neural Networks Using Funnel Regularization

Convolutional neural networks compression with low rank and sparse tensor decompositions

TEC-CNN: Towards Efficient Compressing Convolutional Neural Nets with Low-rank Tensor Decomposition

Sparse Kronecker Canonical Polyadic Decomposition for Convolutional Neural Networks Compression

Low-Rank+Sparse Tensor Compression for Neural Networks

Deep neural network compression by Tucker decomposition with nonlinear response

Convolutional neural networks with low-rank regularization

Accelerating the Low-Rank Decomposed Models

Speeding-up and compression convolutional neural networks by low-rank decomposition without fine-tuning

Hybrid Tensor Decomposition in Neural Network Compression

Reduced storage direct tensor ring decomposition for convolutional neural networks compression

An Accuracy-Preserving Neural Network Compression Via Tucker Decomposition

Compression Method for Convolution Kernel with KCP-FGTD

Tensor Decomposition for Model Reduction in Neural Networks: A Review

Unified Framework for Neural Network Compression via Decomposition and Optimal Rank Selection

Semi-tensor Product-based TensorDecomposition for Neural Network Compression

Holistic CNN Compression Via Low-Rank Decomposition with Knowledge Transfer.

Neural Network Compression Based on Tensor Ring Decomposition

Deep Learning Model Compression with Rank Reduction in Tensor Decomposition.