Abstract:Quantization is of significance for compressing the over-parameterized deep neural models and deploying them on resource-limited devices. Fixed-precision quantization suffers from performance drop due to the limited numerical representation ability. Conversely, mixed-precision quantization (MPQ) is advocated to compress the model effectively by allocating heterogeneous bit-width for layers. MPQ is typically organized into a searching-retraining two-stage process. In this paper, we devise a one-shot training-searching paradigm for mixed-precision model compression. Specifically, in the first stage, all potential bit-width configurations are coupled and thus optimized simultaneously within a set of shared weights. However, our observations reveal a previously unseen and severe bit-width interference phenomenon among highly coupled weights during optimization, leading to considerable performance degradation under a high compression ratio. To tackle this problem, we first design a bit-width scheduler to dynamically freeze the most turbulent bit-width of layers during training, to ensure the rest bit-widths converged properly. Then, taking inspiration from information theory, we present an information distortion mitigation technique to align the behavior of the bad-performing bit-widths to the well-performing ones. In the second stage, an inference-only greedy search scheme is devised to evaluate the goodness of configurations without introducing any additional training costs. Extensive experiments on three representative models and three datasets demonstrate the effectiveness of the proposed method. Code can be available on \href{<a class="link-external link-https" href="https://www.github.com/1hunters/retraining-free-quantization" rel="external noopener nofollow">this https URL</a>}{<a class="link-external link-https" href="https://github.com/1hunters/retraining-free-quantization" rel="external noopener nofollow">this https URL</a>}.

Differential Weight Quantization for Multi-Model Compression

Weight Normalization based Quantization for Deep Neural Network Compression

VecQ: Minimal Loss DNN Model Compression With Vectorized Weight Quantization

DilateQuant: Accurate and Efficient Diffusion Quantization via Weight Dilation

MWQ: Multiscale Wavelet Quantized Neural Networks

A Unified Framework of DNN Weight Pruning and Weight Clustering/Quantization Using ADMM

Progressive DNN Compression: A Key to Achieve Ultra-High Weight Pruning and Quantization Rates using ADMM

DeltaDQ: Ultra-High Delta Compression for Fine-Tuned LLMs via Group-wise Dropout and Separate Quantization

Ahcq: Adaptive Hierarchical Clustering Based Quantization Framework for Deep Neural Networks

Adaptive Layerwise Quantization for Deep Neural Network Compression

Retraining-free Model Quantization via One-Shot Weight-Coupling Learning

Μl2q: an Ultra-Low Loss Quantization Method for DNN Compression

Mixed-precision Deep Neural Network Quantization With Multiple Compression Rates

Distribution-aware Adaptive Multi-bit Quantization

decoupleQ: Towards 2-bit Post-Training Uniform Quantization via decoupling Parameters into Integer and Floating Points

SearchQ: Search-based Fine-Grained Quantization for Data-Free Model Compression

Bit-Quantized-Net: an Effective Method for Compressing Deep Neural Networks.

Weight Fixing Networks

Differentiable Model Compression via Pseudo Quantization Noise

Structured Compression by Weight Encryption for Unstructured Pruning and Quantization

QD-Compressor: a Quantization-based Delta Compression Framework for Deep Neural Networks