Abstract:Most state-of-the-art convolutional neural networks (CNNs) are characterised by excessive parameterisation, leading to a high computational burden. Tensor decomposition has emerged as a model reduction technique for compressing deep neural networks. Previous approaches have predominantly relied on either Tucker decomposition or Canonical Polyadic (CP) decomposition for CNNs. However, CP decomposition exhibits exceptional compression capabilities in comparison to Tucker decomposition, which results in a more pronounced accuracy loss. This paper introduces an efficient model compression method, termed TEC-CNN, designed to achieve significant compression while preserving accuracy levels comparable to those of the original models. In TEC-CNN, convolutional layers are identified to obtain convolutional kernels by analysing given models under the principles of low-rank tensor decomposition, and then, calculating the ranks of convolutional kernels. Furthermore, an efficient decomposition schema for the convolutional kernel is proposed with approximate kernel tensor for reducing parameters. Additionally, a novel format of a convolutional sequence is presented and constructed with a reduced number of parameters to replace the original convolutional layers. Finally, the effectiveness of TEC-CNN is assessed across a range of computer vision tasks. For instance, in CIFAR-100 classification, ResNet18 is compressed to 4.1 MB, while Unext, when applied to image segmentation using the International Skin Imaging Collaboration (ISIC) dataset, is reduced to 3.419 MB. When employed for fire object detection with Yolov7, TEC-CNN achieves a model size reduction of 71.6 MB. Comprehensive experimental results underscore that our approach achieves significant model compression while preserving model performance.

Interpretable Relative Squeezing bottleneck design for compact convolutional neural networks model.

RSNet: A Compact Relative Squeezing Net for Image Recognition

Lightweight Image Super-Resolution Network Using 3D Convolutional Neural Networks

Sensitivity-Oriented Layer-Wise Acceleration and Compression for Convolutional Neural Network.

Focused Quantization for Sparse CNNs

Aggregated Squeeze-and-excitation Transformations for Densely Connected Convolutional Networks

A CNN Compression Method Via Dynamic Channel Ranking Strategy

Learning Efficient Convolutional Networks Through Network Slimming.

S-DenseNet: A DenseNet Compression Model Based on Convolution Grouping Strategy Using Skyline Method.

Residual squeeze CNDS deep learning CNN model for very large scale places image recognition

Exploiting Kernel Sparsity and Entropy for Interpretable CNN Compression

TEC-CNN: Towards Efficient Compressing Convolutional Neural Nets with Low-rank Tensor Decomposition

ChannelNets: Compact and Efficient Convolutional Neural Networks via Channel-Wise Convolutions

SlimConv: Reducing Channel Redundancy in Convolutional Neural Networks by Features Recombining

Towards Compact CNNs via Collaborative Compression

Graph Structure Learning-Based Compression Method for Convolutional Neural Networks.

A Survey of Model Compression and Acceleration for Deep Neural Networks.

Learning Low Resource Consumption CNN through Pruning and Quantization

SCConv: Spatial and Channel Reconstruction Convolution for Feature Redundancy

Sensitivity-based Acceleration and Compression Algorithm for Convolution Neural Network.

Efficient Neural Network Compression Inspired by Compressive Sensing.