Abstract:Convolutional neural networks (CNNs) have achieved remarkable success in various computer vision tasks, which are extremely powerful to deal with massive training data by using tens of millions of parameters. However, CNNs often cost significant memory and computation consumption, which prohibits their usage in resource-limited environments such as mobile or embedded devices. To address the above issues, the existing approaches typically focus on either accelerating the convolutional layers or compressing the fully-connected layers separatedly, without pursuing a joint optimum. In this paper, we overcome such a limitation by introducing a holistic CNN compression framework, termed LRDKT, which works throughout both convolutional and fully-connected layers. First, a low-rank decomposition (LRD) scheme is proposed to remove redundancies across both convolutional kernels and fully-connected matrices, which has a novel closed-form solver to significantly improve the efficiency of the existing iterative optimization solvers. Second, a novel knowledge transfer (KT) based training scheme is introduced. To recover the accumulated accuracy loss and overcome the vanishing gradient, KT explicitly aligns outputs and intermediate responses from a teacher (original) network to its student (compressed) network. We have comprehensively analyzed and evaluated the compression and speedup ratios of the proposed model on MNIST and ILSVRC 2012 benchmarks. In both benchmarks, the proposed scheme has demonstrated superior performance gains over the state-of-the-art methods. We also demonstrate the proposed compression scheme for the task of transfer learning, including domain adaptation and object detection, which show exciting performance gains over the state-of-the-arts. Our source code and compressed models are available at https://github.com/ShaohuiLin/LRDKT.

SmartIdx: Reducing Communication Cost in Federated Learning by Exploiting the CNNs Structures

Exploiting Kernel Sparsity and Entropy for Interpretable CNN Compression

FedComp: A Federated Learning Compression Framework for Resource-Constrained Edge Computing Devices

A Model Compression Method Using Significant Data and Knowledge Distillation

Towards Efficient Network Compression Via Few-Shot Slimming.

Accelerating Convolutional Neural Networks by Removing Interspatial and Interkernel Redundancies.

Efficient Model Compression via Global Sparsification for Over-the-Air Federated Learning.

Focused Quantization for Sparse CNNs

Learning Efficient Convolutional Networks Through Network Slimming.

SlimConv: Reducing Channel Redundancy in Convolutional Neural Networks by Features Recombining

Towards Compact CNNs via Collaborative Compression

Holistic CNN Compression Via Low-Rank Decomposition with Knowledge Transfer.

Low-Rate Feature Compression for Collaborative Intelligence: Reducing Redundancy in Spatial and Statistical Levels

Reduced storage direct tensor ring decomposition for convolutional neural networks compression

Iterative Deep Model Compression and Acceleration in the Frequency Domain.

Layer-Wise Training To Create Efficient Convolutional Neural Networks

Reducing Communication for Split Learning by Randomized Top-k Sparsification

An Efficient CNN Inference Accelerator Based on Intra- and Inter-Channel Feature Map Compression

Deep Neural Network Interlayer Feature Map Compression Based on Least-Squares Fitting.

Deep Convolutional Neural Networks Compression Method Based on Linear Representation of Kernels

Efficient Wireless Federated Learning via Low-Rank Gradient Factorization