Abstract:Most state-of-the-art convolutional neural networks (CNNs) are characterised by excessive parameterisation, leading to a high computational burden. Tensor decomposition has emerged as a model reduction technique for compressing deep neural networks. Previous approaches have predominantly relied on either Tucker decomposition or Canonical Polyadic (CP) decomposition for CNNs. However, CP decomposition exhibits exceptional compression capabilities in comparison to Tucker decomposition, which results in a more pronounced accuracy loss. This paper introduces an efficient model compression method, termed TEC-CNN, designed to achieve significant compression while preserving accuracy levels comparable to those of the original models. In TEC-CNN, convolutional layers are identified to obtain convolutional kernels by analysing given models under the principles of low-rank tensor decomposition, and then, calculating the ranks of convolutional kernels. Furthermore, an efficient decomposition schema for the convolutional kernel is proposed with approximate kernel tensor for reducing parameters. Additionally, a novel format of a convolutional sequence is presented and constructed with a reduced number of parameters to replace the original convolutional layers. Finally, the effectiveness of TEC-CNN is assessed across a range of computer vision tasks. For instance, in CIFAR-100 classification, ResNet18 is compressed to 4.1 MB, while Unext, when applied to image segmentation using the International Skin Imaging Collaboration (ISIC) dataset, is reduced to 3.419 MB. When employed for fire object detection with Yolov7, TEC-CNN achieves a model size reduction of 71.6 MB. Comprehensive experimental results underscore that our approach achieves significant model compression while preserving model performance.

Research on Network Layer Recursive Reduction Model Compression for Image Recognition

Improved Model Compression Method Based on Information Entropy

A Model Compression Method Using Significant Data and Knowledge Distillation

Deep Learning Model Compression with Rank Reduction in Tensor Decomposition.

Incremental Layers Resection: A Novel Method to Compress Neural Networks

A Novel Deep Learning Model Compression Algorithm

Sensitivity-Oriented Layer-Wise Acceleration and Compression for Convolutional Neural Network.

Sensitiveness Based Strategy for Network Compression

CNN Compression-Recovery Framework Via Rank Allocation Decomposition with Knowledge Transfer

Filter Pruning with a Feature Map Entropy Importance Criterion for Convolution Neural Networks Compressing

Neural Network Compression Via Effective Filter Analysis and Hierarchical Pruning

Layer-Wise Training To Create Efficient Convolutional Neural Networks

ResNet Structure Simplification with the Convolutional Kernel Redundancy Measure

TEC-CNN: Towards Efficient Compressing Convolutional Neural Nets with Low-rank Tensor Decomposition

Layer Pruning via Fusible Residual Convolutional Block for Deep Neural Networks

Deep neural network pruning method based on sensitive layers and reinforcement learning

A CNN Compression Method Via Dynamic Channel Ranking Strategy

Two-Stage Model Compression and Acceleration: Optimal Student Network for Better Performance

Learning Low Resource Consumption CNN through Pruning and Quantization

Learning Efficient Convolutional Networks Through Network Slimming.

On the importance of network architecture in training very deep neural networks