Abstract:Convolutional neural networks (CNNs) have achieved remarkable success in various computer vision tasks, which are extremely powerful to deal with massive training data by using tens of millions of parameters. However, CNNs often cost significant memory and computation consumption, which prohibits their usage in resource-limited environments such as mobile or embedded devices. To address the above issues, the existing approaches typically focus on either accelerating the convolutional layers or compressing the fully-connected layers separatedly, without pursuing a joint optimum. In this paper, we overcome such a limitation by introducing a holistic CNN compression framework, termed LRDKT, which works throughout both convolutional and fully-connected layers. First, a low-rank decomposition (LRD) scheme is proposed to remove redundancies across both convolutional kernels and fully-connected matrices, which has a novel closed-form solver to significantly improve the efficiency of the existing iterative optimization solvers. Second, a novel knowledge transfer (KT) based training scheme is introduced. To recover the accumulated accuracy loss and overcome the vanishing gradient, KT explicitly aligns outputs and intermediate responses from a teacher (original) network to its student (compressed) network. We have comprehensively analyzed and evaluated the compression and speedup ratios of the proposed model on MNIST and ILSVRC 2012 benchmarks. In both benchmarks, the proposed scheme has demonstrated superior performance gains over the state-of-the-art methods. We also demonstrate the proposed compression scheme for the task of transfer learning, including domain adaptation and object detection, which show exciting performance gains over the state-of-the-arts. Our source code and compressed models are available at https://github.com/ShaohuiLin/LRDKT.

Learning from the NN-based Compressed Domain with Deep Feature Reconstruction Loss

EiCSNet: Efficient Iterative Neural Network for Compressed Sensing Reconstruction

Image reconstruction based on back propagation learning in Compressed Sensing theory

DNN-Compressed Domain Visual Recognition with Feature Adaptation

Compressing complex convolutional neural network based on an improved deep compression algorithm

Near-lossless L-infinity Constrained Multi-rate Image Decompression Via Deep Neural Network

Compressed Domain Image Classification Using a Dynamic-Rate Neural Network

Classification Guided Deep Convolutional Network For Compressed Sensing

Deep Image Compression via End-to-End Learning

2C-Net: integrate image compression and classification via deep neural network

Image compression optimized for 3D reconstruction by utilizing deep neural networks

A Deep Image Compression Framework for Face Recognition

Unified Architecture Adaptation for Compressed Domain Semantic Inference

Deep Learning Model Compression with Rank Reduction in Tensor Decomposition.

Research on Image Reconstruction of Compressed Sensing Based on a Multi-Feature Residual Network

Near-Lossless ℓ∞-Constrained Image Decompression Via Deep Neural Network.

Holistic CNN Compression Via Low-Rank Decomposition with Knowledge Transfer.

Mixed-Resolution Image Representation and Compression with Convolutional Neural Networks.

Reconstruction-free Image Compression for Machine Vision via Knowledge Transfer

Analysis on Compressed Domain: A Multi-Task Learning Approach

Compressed Network in Network Models for Traffic Classification