Abstract:While previous network compression methods achieve great success, most of them rely on the abundant training data which is, unfortunately, often unavailable in practice due to some reasons, e.g., privacy issues, storage constraints, and transmission limitations. A promising way to solve this problem is to perform compression with a few unlabeled data. Proceeding along this way, we propose a novel few-shot network compression framework named Few-Shot Slimming (FSS). FSS follows the student/teacher paradigm, and contains two steps: (1) construct the student by inheriting principal feature maps from the teacher; (2) refine the student feature representation by knowledge distillation with an enhanced mixing data augmentation method called GridMix. Specifically, in the first step, we employ normalized cross correlation to perform the principal feature analysis, and then theoretically construct a new indicator to select the most informative feature maps from the teacher for the student. The indicator is based on the variances of feature maps which can efficiently quantitate the information richness of the input feature maps in a feature-agnostic manner. In the second step, we perform the knowledge distillation for the initialized student in first step with a novel grid-based mixing data augmentation technique which greatly extends the limited sample dataset. In this way, the student is able to refine its feature representation and achieves a better result. Extensive experiments on multiple benchmarks demonstrate the state-of-the-art performance of FSS. For example, by using 0.2% label-free data of full training set, FSS yields a 60% FLOPs reduction for DenseNet-40 on CIFAR-10 with only a loss of 0.8% in top-1 accuracy, achieving a result on par with that obtained by the conventional full-data methods.

Optimization Feature Compression and Fnn Realization

Registration and stitching optimization for two-scene-type remote sensing image

Discrete K-L Transform And Information Feature Analysis

TECO: A Unified Feature Map Compression Framework Based on Transform and Entropy

Towards Efficient Network Compression Via Few-Shot Slimming.

A feature compression method based on similarity matching

Fast hybrid fractal image compression using an image feature and neural network

MODIFIED KERNEL-BASED NONLINEAR FEATURE EXTRACTION

AFEC: Adaptive Feature Extraction Modules for Learned Image Compression

Entropy Optimized Deep Feature Compression.

DMOFC: Discrimination Metric-Optimized Feature Compression

Comprehensive SNN Compression Using ADMM Optimization and Activity Regularization

2C-Net: integrate image compression and classification via deep neural network

Towards Analysis-Friendly Face Representation with Scalable Feature and Texture Compression

Toward Intelligent Sensing: Intermediate Deep Feature Compression

Exploiting Kernel Sparsity and Entropy for Interpretable CNN Compression

Low Rank Optimization for Efficient Deep Learning: Making A Balance between Compact Architecture and Fast Training

Neural Network Compression Via Sparse Optimization

Efficient Neural Network Compression Inspired by Compressive Sensing.

Finesse: Fine-Grained Feature Locality Based Fast Resemblance Detection For Post-Deduplication Delta Compression

Codedretrieval: Joint Image Compression and Retrieval with Neural Networks.