Abstract:Convolutional neural networks (CNNs) possess strong expressive capabilities, but with the increasing size and cost of modern CNNs, deploying large-scale CNNs in resource-limited environments remains a challenge, making neural network compression a hot topic in contemporary research. Among various compression methods, tensor decomposition shows numerous unique advantages, such as regular data structure, solid mathematical foundation, convenience in training from scratch, and ideal compression rates. However, most existing tensor decompositions do not perform well in compressing CNNs, struggling to balance time and space complexity, and failing to accelerate CNNs in reality. To address these issues. In this paper, we propose a fine-grained KCP tensor decomposition for CNN convolution kernel compression with considerable compression rate and acceleration capability. First, convolution kernels are compressed using 2nd-order KCP decomposition. The decomposed factor tensors are then converted into a four-layer lightweight convolution kernel structure. This structure conducts continuous contraction computations with the input, transforming the convolution computations into a highly efficient lightweight layered matrix multiplication structure. Secondly, we conduct a comprehensive comparison with other methods and analyze the superiority of our method. Comparisons of computational complexity with other typical tensor compression methods, along with simulation experiments on CIFAR-10 and ImageNet datasets, demonstrate that our method can achieve more efficient comprehensive results. These include acceptable accuracy, significantly reduced model size, and real-world acceleration of network inference times. Finally, we further analyze how different combinations of KT rank and CP rank impact the compression effectiveness and expressive capacity of CNNs.

Accelerating Convolutional Neural Networks With Dominant Convolutional Kernel And Knowledge Pre-Regression

A Model Compression Method Using Significant Data and Knowledge Distillation

Sensitivity-Oriented Layer-Wise Acceleration and Compression for Convolutional Neural Network.

Accelerating Convolutional Neural Networks by Removing Interspatial and Interkernel Redundancies.

Holistic CNN Compression Via Low-Rank Decomposition with Knowledge Transfer.

A Survey of Model Compression and Acceleration for Deep Neural Networks.

Accelerating Convolutional Neural Networks Using Fine-Tuned Backpropagation Progress

Sparse Kronecker Canonical Polyadic Decomposition for Convolutional Neural Networks Compression

An efficient approach to escalate the speed of training convolution neural networks

Sensitivity-based Acceleration and Compression Algorithm for Convolution Neural Network.

Deep Convolutional Neural Networks Compression Method Based on Linear Representation of Kernels

Two-Stage Model Compression and Acceleration: Optimal Student Network for Better Performance

Analysis of Model Compression Using Knowledge Distillation

Model Selection - Knowledge Distillation Framework for Model Compression

Compression Method for Convolution Kernel with KCP-FGTD

Hierarchical Knowledge Squeezed Adversarial Network Compression

CNN Compression-Recovery Framework Via Rank Allocation Decomposition with Knowledge Transfer

Accelerating Convolutional Neural Networks via Activation Map Compression

Deep Learning Model Compression with Rank Reduction in Tensor Decomposition.

A Pre-defined Sparse Kernel Based Convolution for Deep CNNs

Model Compression via Collaborative Data-Free Knowledge Distillation for Edge Intelligence.