Abstract:While previous network compression methods achieve great success, most of them rely on the abundant training data which is, unfortunately, often unavailable in practice due to some reasons, e.g., privacy issues, storage constraints, and transmission limitations. A promising way to solve this problem is to perform compression with a few unlabeled data. Proceeding along this way, we propose a novel few-shot network compression framework named Few-Shot Slimming (FSS). FSS follows the student/teacher paradigm, and contains two steps: (1) construct the student by inheriting principal feature maps from the teacher; (2) refine the student feature representation by knowledge distillation with an enhanced mixing data augmentation method called GridMix. Specifically, in the first step, we employ normalized cross correlation to perform the principal feature analysis, and then theoretically construct a new indicator to select the most informative feature maps from the teacher for the student. The indicator is based on the variances of feature maps which can efficiently quantitate the information richness of the input feature maps in a feature-agnostic manner. In the second step, we perform the knowledge distillation for the initialized student in first step with a novel grid-based mixing data augmentation technique which greatly extends the limited sample dataset. In this way, the student is able to refine its feature representation and achieves a better result. Extensive experiments on multiple benchmarks demonstrate the state-of-the-art performance of FSS. For example, by using 0.2% label-free data of full training set, FSS yields a 60% FLOPs reduction for DenseNet-40 on CIFAR-10 with only a loss of 0.8% in top-1 accuracy, achieving a result on par with that obtained by the conventional full-data methods.

CondenseNet V2: Sparse Feature Reactivation for Deep Networks

Deep Neural Network Acceleration with Sparse Prediction Layers

Condense: A Framework for Device and Frequency Adaptive Neural Network Models on the Edge.

DCCD: Reducing Neural Network Redundancy Via Distillation

Pruning by Training: A Novel Deep Neural Network Compression Framework for Image Processing.

CondenseNet: An Efficient DenseNet Using Learned Group Convolutions

Towards Efficient Network Compression Via Few-Shot Slimming.

SparseNet: A Sparse DenseNet for Image Classification

SlimConv: Reducing Channel Redundancy in Convolutional Neural Networks by Features Recombining

Efficient Network Compression Through Smooth-Lasso Constraint

Efficient and Flexible Method for Reducing Moderate-size Deep Neural Networks with Condensation

Neural Network Compression Via Sparse Optimization

Efficient Neural Network Compression Inspired by Compressive Sensing.

Connection Reduction of DenseNet for Image Recognition

Neural Network Compression by Joint Sparsity Promotion and Redundancy Reduction

Multi-scale Convolution Aggregation and Stochastic Feature Reuse for DenseNets

SparseNN: A Performance-Efficient Accelerator for Large-Scale Sparse Neural Networks

Prune the Convolutional Neural Networks with Sparse Shrink

Dual sparse training framework: inducing activation map sparsity via Transformed $\ell1$ regularization

Squeeze, Recover and Relabel: Dataset Condensation at ImageNet Scale From A New Perspective

Pre-defined Sparsity for Low-Complexity Convolutional Neural Networks