Abstract:While previous network compression methods achieve great success, most of them rely on the abundant training data which is, unfortunately, often unavailable in practice due to some reasons, e.g., privacy issues, storage constraints, and transmission limitations. A promising way to solve this problem is to perform compression with a few unlabeled data. Proceeding along this way, we propose a novel few-shot network compression framework named Few-Shot Slimming (FSS). FSS follows the student/teacher paradigm, and contains two steps: (1) construct the student by inheriting principal feature maps from the teacher; (2) refine the student feature representation by knowledge distillation with an enhanced mixing data augmentation method called GridMix. Specifically, in the first step, we employ normalized cross correlation to perform the principal feature analysis, and then theoretically construct a new indicator to select the most informative feature maps from the teacher for the student. The indicator is based on the variances of feature maps which can efficiently quantitate the information richness of the input feature maps in a feature-agnostic manner. In the second step, we perform the knowledge distillation for the initialized student in first step with a novel grid-based mixing data augmentation technique which greatly extends the limited sample dataset. In this way, the student is able to refine its feature representation and achieves a better result. Extensive experiments on multiple benchmarks demonstrate the state-of-the-art performance of FSS. For example, by using 0.2% label-free data of full training set, FSS yields a 60% FLOPs reduction for DenseNet-40 on CIFAR-10 with only a loss of 0.8% in top-1 accuracy, achieving a result on par with that obtained by the conventional full-data methods.

Semantic-Oriented Feature Compression for Resource-Constrained Distributed Sensor Networks

Compressive Sensing Based Asymmetric Semantic Image Compression for Resource-Constrained IoT System

Towards Efficient Network Compression Via Few-Shot Slimming.

Dual-branch spectral–spatial feature extraction network for multispectral image compression

Semantic Scalable Image Compression with Cross-Layer Priors.

Joint Task and Data Oriented Semantic Communications: A Deep Separate Source-channel Coding Scheme

Unified Architecture Adaptation for Compressed Domain Semantic Inference

Towards Semantically Scalable Image Coding Using Semantic Map.

Digital-SC: Digital Semantic Communication with Adaptive Network Split and Learned Non-Linear Quantization

Semantic Ensemble Loss and Latent Refinement for High-Fidelity Neural Image Compression

Classification-Driven Discrete Neural Representation Learning for Semantic Communications

Semantic redundancy-aware implicit neural compression for multidimensional biomedical image data

Supervised Compression for Resource-Constrained Edge Computing Systems

SSSIC: Semantics-to-Signal Scalable Image Coding with Learned Structural Representations.

SNN-SC: A Spiking Semantic Communication Framework for Collaborative Intelligence

Rethinking Semantic Image Compression: Scalable Representation with Cross-modality Transfer

Self-Supervised Scalable Deep Compressed Sensing

SQuantizer: Simultaneous Learning for Both Sparse and Low-precision Neural Networks

DeepSIC: Deep Semantic Image Compression