Abstract:While previous network compression methods achieve great success, most of them rely on the abundant training data which is, unfortunately, often unavailable in practice due to some reasons, e.g., privacy issues, storage constraints, and transmission limitations. A promising way to solve this problem is to perform compression with a few unlabeled data. Proceeding along this way, we propose a novel few-shot network compression framework named Few-Shot Slimming (FSS). FSS follows the student/teacher paradigm, and contains two steps: (1) construct the student by inheriting principal feature maps from the teacher; (2) refine the student feature representation by knowledge distillation with an enhanced mixing data augmentation method called GridMix. Specifically, in the first step, we employ normalized cross correlation to perform the principal feature analysis, and then theoretically construct a new indicator to select the most informative feature maps from the teacher for the student. The indicator is based on the variances of feature maps which can efficiently quantitate the information richness of the input feature maps in a feature-agnostic manner. In the second step, we perform the knowledge distillation for the initialized student in first step with a novel grid-based mixing data augmentation technique which greatly extends the limited sample dataset. In this way, the student is able to refine its feature representation and achieves a better result. Extensive experiments on multiple benchmarks demonstrate the state-of-the-art performance of FSS. For example, by using 0.2% label-free data of full training set, FSS yields a 60% FLOPs reduction for DenseNet-40 on CIFAR-10 with only a loss of 0.8% in top-1 accuracy, achieving a result on par with that obtained by the conventional full-data methods.

S-DenseNet: A DenseNet Compression Model Based on Convolution Grouping Strategy Using Skyline Method.

Automl For Densenet Compression

Towards Efficient Network Compression Via Few-Shot Slimming.

Efficient densely connected convolutional neural networks

Sensitivity-Oriented Layer-Wise Acceleration and Compression for Convolutional Neural Network.

Efficient Network Compression Through Smooth-Lasso Constraint

Aggregated Squeeze-and-excitation Transformations for Densely Connected Convolutional Networks

SparseNet: A Sparse DenseNet for Image Classification

Two-Stage Model Compression and Acceleration: Optimal Student Network for Better Performance

CED-Net: A more effective DenseNet model with channel enhancement

CondenseNet: An Efficient DenseNet Using Learned Group Convolutions

On Compressing Deep Models by Low Rank and Sparse Decomposition.

Interpretable Relative Squeezing bottleneck design for compact convolutional neural networks model.

Convolutional Networks with Dense Connectivity

Connection Reduction of DenseNet for Image Recognition

Efficient Neural Network Compression Inspired by Compressive Sensing.

Learning Efficient Convolutional Networks Through Network Slimming.

A CNN Compression Method Via Dynamic Channel Ranking Strategy

SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <1MB model size

SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size

ChannelNets: Compact and Efficient Convolutional Neural Networks via Channel-Wise Convolutions