Abstract:This paper presents incremental network quantization (INQ), a novel method, targeting to efficiently convert any pre-trained full-precision convolutional neural network (CNN) model into a low-precision version whose weights are constrained to be either powers of two or zero. Unlike existing methods which are struggled in noticeable accuracy loss, our INQ has the potential to resolve this issue, as benefiting from two innovations. On one hand, we introduce three interdependent operations, namely weight partition, group-wise quantization and re-training. A well-proven measure is employed to divide the weights in each layer of a pre-trained CNN model into two disjoint groups. The weights in the first group are responsible to form a low-precision base, thus they are quantized by a variable-length encoding method. The weights in the other group are responsible to compensate for the accuracy loss from the quantization, thus they are the ones to be re-trained. On the other hand, these three operations are repeated on the latest re-trained group in an iterative manner until all the weights are converted into low-precision ones, acting as an incremental network quantization and accuracy enhancement procedure. Extensive experiments on the ImageNet classification task using almost all known deep CNN architectures including AlexNet, VGG-16, GoogleNet and ResNets well testify the efficacy of the proposed method. Specifically, at 5-bit quantization, our models have improved accuracy than the 32-bit floating-point references. Taking ResNet-18 as an example, we further show that our quantized models with 4-bit, 3-bit and 2-bit ternary weights have improved or very similar accuracy against its 32-bit floating-point baseline. Besides, impressive results with the combination of network pruning and INQ are also reported. The code is available at https://github.com/Zhouaojun/Incremental-Network-Quantization.

SeerNet: Predicting Convolutional Neural Network Feature-Map Sparsity Through Low-Bit Quantization.

Deep Neural Network Acceleration with Sparse Prediction Layers

Focused Quantization for Sparse CNNs

Low-precision CNN Model Quantization based on Optimal Scaling Factor Estimation

Exploring Fine-Grained Sparsity in Convolutional Neural Networks for Efficient Inference

Learning Low Resource Consumption CNN through Pruning and Quantization

SQuantizer: Simultaneous Learning for Both Sparse and Low-precision Neural Networks

Exploring the Granularity of Sparsity in Convolutional Neural Networks.

Accelerating Neural Network Inference by Overflow Aware Quantization

Exploiting Kernel Sparsity and Entropy for Interpretable CNN Compression

Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights

KCNN: Kernel-wise Quantization to Remarkably Decrease Multiplications in Convolutional Neural Network.

Efficient Neural Network Compression Inspired by Compressive Sensing.

EBSP: evolving bit sparsity patterns for hardware-friendly inference of quantized deep neural networks

Mixed-precision quantized neural networks with progressively decreasing bitwidth

Mixed-Precision Quantized Neural Network with Progressively Decreasing Bitwidth For Image Classification and Object Detection.

FactorizeNet: Progressive Depth Factorization for Efficient Network Architecture Exploration Under Quantization Constraints

Efficient Neural Networks with Spatial Wise Sparsity Using Unified Importance Map.

Fixed-point Quantization of Convolutional Neural Networks for Quantized Inference on Embedded Platforms

Pyramid Vector Quantization and Bit Level Sparsity in Weights for Efficient Neural Networks Inference

General Bitwidth Assignment for Efficient Deep Convolutional Neural Network Quantization