Abstract:Deep convolutional neural networks have a large number of parameters and require a significant number of floating-point operations during computation, which limits their deployment in situations where the storage space is limited and computational resources are insufficient, such as in mobile phones and small robots. Many network compression methods have been proposed to address the aforementioned issues, including pruning, low-rank decomposition, quantization, etc. However, these methods typically fail to achieve a significant compression ratio in terms of the parameter count. Even when high compression rates are achieved, the network's performance is often significantly deteriorated, making it difficult to perform tasks effectively. In this study, we propose a more compact representation for neural networks, named Quantized Low-Rank Tensor Decomposition (QLTD), to super compress deep convolutional neural networks. Firstly, we employed low-rank Tucker decomposition to compress the pre-trained weights. Subsequently, to further exploit redundancies within the core tensor and factor matrices obtained through Tucker decomposition, we employed vector quantization to partition and cluster the weights. Simultaneously, we introduced a self-attention module for each core tensor and factor matrix to enhance the training responsiveness in critical regions. The object identification results in the CIFAR10 experiment showed that QLTD achieved a compression ratio of 35.43×, with less than 1% loss in accuracy and a compression ratio of 90.61×, with less than a 2% loss in accuracy. QLTD was able to achieve a significant compression ratio in terms of the parameter count and realize a good balance between compressing parameters and maintaining identification accuracy.

Compression algorithm for live face recognition model based on depth-separable convolution

A Deep Image Compression Framework for Face Recognition

Compressing Deep Convolutional Networks using Vector Quantization

Tightly-Coupled Data Compression for Efficient Face Alignment

Design Light-weight 3D Convolutional Networks for Video Recognition Temporal Residual, Fully Separable Block, and Fast Algorithm

Face Recognition Based on Lightweight Convolutional Neural Networks

Mitigate Compression Artifacts for Face in Video Recognition

Compressing complex convolutional neural network based on an improved deep compression algorithm

Deep learning model compression using network sensitivity and gradients

Sensitivity-based Acceleration and Compression Algorithm for Convolution Neural Network.

Ef-QuantFace: Streamlined Face Recognition with Small Data and Low-Bit Precision

Content-aware Facial Image Compression with Deep Learning Method

Focused Quantization for Sparse CNNs

Towards Super Compressed Neural Networks for Object Identification: Quantized Low-Rank Tensor Decomposition with Self-Attention

Lightweight Compression of Intermediate Neural Network Features for Collaborative Intelligence

Convolutional Neural Network Compression Based on Low-Rank Decomposition

Improved deep learning image compression model: performance optimization based on convolutional modules and local attention mechanism

Facial expression recognition based on improved depthwise separable convolutional network

Improved JPEG Lossless Compression for Compression of Intermediate Layers in Neural Networks Based on Compute-In-Memory

Iterative Deep Model Compression and Acceleration in the Frequency Domain.

Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding