Abstract:Deep convolutional neural networks have a large number of parameters and require a significant number of floating-point operations during computation, which limits their deployment in situations where the storage space is limited and computational resources are insufficient, such as in mobile phones and small robots. Many network compression methods have been proposed to address the aforementioned issues, including pruning, low-rank decomposition, quantization, etc. However, these methods typically fail to achieve a significant compression ratio in terms of the parameter count. Even when high compression rates are achieved, the network's performance is often significantly deteriorated, making it difficult to perform tasks effectively. In this study, we propose a more compact representation for neural networks, named Quantized Low-Rank Tensor Decomposition (QLTD), to super compress deep convolutional neural networks. Firstly, we employed low-rank Tucker decomposition to compress the pre-trained weights. Subsequently, to further exploit redundancies within the core tensor and factor matrices obtained through Tucker decomposition, we employed vector quantization to partition and cluster the weights. Simultaneously, we introduced a self-attention module for each core tensor and factor matrix to enhance the training responsiveness in critical regions. The object identification results in the CIFAR10 experiment showed that QLTD achieved a compression ratio of 35.43×, with less than 1% loss in accuracy and a compression ratio of 90.61×, with less than a 2% loss in accuracy. QLTD was able to achieve a significant compression ratio in terms of the parameter count and realize a good balance between compressing parameters and maintaining identification accuracy.

A Robust Low-Rank Tensor Decomposition and Quantization Based Compression Method

Tensor Denoising Using Low-Rank Tensor Train Decomposition

Decomposition of Big Tensors With Low Multilinear Rank

A Low-Rank and Sparse Enhanced Tucker Decomposition Approach for Tensor Completion

An Iterative Reweighted Method for Tucker Decomposition of Incomplete Multiway Tensors

Low Tucker rank tensor completion using a symmetric block coordinate descent method

Robust Low-rank Tensor Decomposition with the L2 Criterion

Static and Streaming Tucker Decomposition for Dense Tensors

Semi-tensor Product-based TensorDecomposition for Neural Network Compression

Deep neural network compression by Tucker decomposition with nonlinear response

On Optimizing Distributed Tucker Decomposition for Dense Tensors

An Accuracy-Preserving Neural Network Compression Via Tucker Decomposition

Hybrid Tensor Decomposition in Neural Network Compression

TensorCodec: Compact Lossy Compression of Tensors without Strong Data Assumptions

Low-Rank and Sparse Enhanced Tucker Decomposition for Tensor Completion

Low Rank Triple Decomposition and Tensor Recovery

BATUDE: Budget-Aware Neural Network Compression Based on Tucker Decomposition

Low-Rank+Sparse Tensor Compression for Neural Networks

Low-rank optimization on Tucker tensor varieties

Towards Super Compressed Neural Networks for Object Identification: Quantized Low-Rank Tensor Decomposition with Self-Attention

Stable Low-rank Tensor Decomposition for Compression of Convolutional Neural Network