Abstract:Network pruning and quantization are proven to be effective ways for deep model compression. To obtain a highly compact model, most methods first perform network pruning and then conduct quantization based on the pruned model. However, this strategy may ignore that the pruning and quantization would affect each other and thus performing them separately may lead to sub-optimal performance. To address this, performing pruning and quantization jointly is essential. Nevertheless, how to make a trade-off between pruning and quantization is non-trivial. Moreover, existing compression methods often rely on some pre-defined compression configurations (i.e., pruning rates or bitwidths). Some attempts have been made to search for optimal configurations, which however may take unbearable optimization cost. To address these issues, we devise a simple yet effective method named Single-path Bit Sharing (SBS) for automatic loss-aware model compression. To this end, we consider the network pruning as a special case of quantization and provide a unified view for model pruning and quantization. We then introduce a single-path model to encode all candidate compression configurations, where a high bitwidth value will be decomposed into the sum of a lowest bitwidth value and a series of re-assignment offsets. Relying on the single-path model, we introduce learnable binary gates to encode the choice of configurations and learn the binary gates and model parameters jointly. More importantly, the configuration search problem can be transformed into a subset selection problem, which helps to significantly reduce the optimization difficulty and computation cost. In this way, the compression configurations of each layer and the trade-off between pruning and quantization can be automatically determined. Extensive experiments on CIFAR-100 and ImageNet show that SBS significantly reduces computation cost while achieving promising performance. For example, our SBS compressed MobileNetV2 achieves 22.6× Bit-Operation (BOP) reduction with only 0.1% drop in the Top-1 accuracy.

Sensitivity-Aware Bit Allocation for Intermediate Deep Feature Compression.

Intermediate Deep Feature Compression: the Next Battlefield of Intelligent Sensing

Toward Intelligent Sensing: Intermediate Deep Feature Compression

Agnostic Feature Compression with Semantic Guided Channel Importance Analysis

Saliency Segmentation Oriented Deep Image Compression with Novel Bit Allocation

Lossy Intermediate Deep Learning Feature Compression and Evaluation

Sensitivity Decouple Learning for Image Compression Artifacts Reduction

ASC: Adaptive Scale Feature Map Compression for Deep Neural Network

A feature compression method based on similarity matching

Private Compression for Intermediate Feature in Iot-Supported Mobile Cloud Inference

End-to-End Image Compression Via Attention-Guided Information-Preserving Module

An End-to-End Channel-Adaptive Feature Compression Approach in Device-Edge Co-Inference Systems

LBS: Loss-aware Bit Sharing for Automatic Model Compression

Single-path Bit Sharing for Automatic Loss-aware Model Compression

Content-Based Hyperspectral Image Compression Using a Multi-Depth Weighted Map With Dynamic Receptive Field Convolution

Region-of-interest and channel attention-based joint optimization of image compression and computer vision

Variable-Rate Deep Image Compression through Spatially-Adaptive Feature Transform

Dual-branch spectral–spatial feature extraction network for multispectral image compression

Learned Image Compression Using Cross-Component Attention Mechanism

Towards Extreme Image Compression with Latent Feature Guidance and Diffusion Prior