Abstract:As the modern deep neural networks (DNNs) have become more and more large-scale and expensive, the topic of DNN compression grows into a hot direction nowadays. Among variant compression methods, tensor decomposition seems to be the most promising and low-cost one because of its solid mathematical foundations and regular data structure. However, most of the existing tensor decompositions are not very good at accelerating DNNs, because there are always necessary transpositions on tensor modes to make the input data calculate with the decomposed factor tensors correctly, and transposition will bring extra memory and time cost for the realistic system without doubt. In this paper, we select a relatively novel Kronecker CANDECOMP/PARAFAC (KCP) tensor decomposition which has fine-grained factor tensors, and propose the transposition-free algorithm to calculate the contractions between the input data and the neural weight in KCP format. The theoretically analysis of computation complexity indicates that the proposed method is much more efficient than the existing algorithms. We further prove that the training complexity of KCP-DNN based on the proposed transposition-free algorithm can also be faster than the traditional ones, and make a comprehensive comparison of space and computation complexity including training and inference stages to show the superiority of our method. As a series of related works pay more attention to the recurrent neural networks (RNNs), we follow these existing practices and focus on the KCP-RNN to make a comprehensive comparison with them, and the experimental results show our KCP-RNN with transposition-free algorithm has systematically advantages including accuracy, space complexity, computation complexity, and realistic running time. Besides, some advanced characteristics of KCP-DNN such as collocation of ranks, have also been discussed.

Speeding Up Deep Convolutional Neural Networks Based on Tucker-CP Decomposition

MLCNN: Cross-Layer Cooperative Optimization and Accelerator Architecture for Speeding Up Deep Learning Applications

Deep neural network compression by Tucker decomposition with nonlinear response

TDC: Towards Extremely Efficient CNNs on GPUs via Hardware-Aware Tucker Decomposition

TEC-CNN: Towards Efficient Compressing Convolutional Neural Nets with Low-rank Tensor Decomposition

Sparse Kronecker Canonical Polyadic Decomposition for Convolutional Neural Networks Compression

Compression Method for Convolution Kernel with KCP-FGTD

Speeding-up and compression convolutional neural networks by low-rank decomposition without fine-tuning

Convolution Without Multiplication: A General Speed Up Strategy for CNNs

Accelerating Training using Tensor Decomposition

Fast CP‐compression Layer: Tensor CP‐decomposition to Compress Layers in Deep Learning

CNN Acceleration by Low-rank Approximation with Quantized Factors

Sensitivity-Oriented Layer-Wise Acceleration and Compression for Convolutional Neural Network.

Mixed-TD: Efficient Neural Network Accelerator with Layer-Specific Tensor Decomposition

Sparsing Deep Neural Network Using Semi-Discrete Matrix Decomposition

Remote Sensing Imagery Object Detection Model Compression via Tucker Decomposition

Realistic Acceleration of Neural Networks with Fine-Grained Tensor Decomposition

Depth-wise Decomposition for Accelerating Separable Convolutions in Efficient Convolutional Neural Networks

Accelerating Very Deep Convolutional Networks for Classification and Detection

Speeding up Resnet Architecture with Layers Targeted Low Rank Decomposition

An Accuracy-Preserving Neural Network Compression Via Tucker Decomposition