Abstract:The deep layers of modern neural networks extract a rather rich set of features as an input propagates through the network. This paper sets out to harvest these rich intermediate representations for quantization with minimal accuracy loss while significantly reducing the memory footprint and compute intensity of the DNN. This paper utilizes knowledge distillation through teacher-student paradigm (Hinton et al., 2015) in a novel setting that exploits the feature extraction capability of DNNs for higher-accuracy quantization. As such, our algorithm logically divides a pretrained full-precision DNN to multiple sections, each of which exposes intermediate features to train a team of students independently in the quantized domain. This divide and conquer strategy, in fact, makes the training of each student section possible in isolation while all these independently trained sections are later stitched together to form the equivalent fully quantized network. Our algorithm is a sectional approach towards knowledge distillation and is not treating the intermediate representation as a hint for pretraining before one knowledge distillation pass over the entire network (Romero et al., 2015). Experiments on various DNNs (AlexNet, LeNet, MobileNet, ResNet-18, ResNet-20, SVHN and VGG-11) show that, this approach -- called DCQ (Divide and Conquer Quantization) -- on average, improves the performance of a state-of-the-art quantized training technique, DoReFa-Net (Zhou et al., 2016) by 21.6% and 9.3% for binary and ternary quantization, respectively. Additionally, we show that incorporating DCQ to existing quantized training methods leads to improved accuracies as compared to previously reported by multiple state-of-the-art quantized training methods.

Initialization Seeds Facilitating Neural Network Quantization

Pse: Mixed Quantization Framework of Neural Networks for Efficient Deployment

Optimizing Quantized Neural Networks in a Weak Curvature Manifold

Residual Quantization for Low Bit-Width Neural Networks.

How to Initialize your Network? Robust Initialization for WeightNorm & ResNets

Quantization Networks

AutoQ: Automated Kernel-Wise Neural Network Quantization

Adaptive Quantization for Deep Neural Network

Where Should We Begin? A Low-Level Exploration of Weight Initialization Impact on Quantized Behaviour of Deep Neural Networks

Designing strong baselines for ternary neural network quantization through support and mass equalization

Instance-Aware Dynamic Neural Network Quantization

Post-Training Quantization for Re-parameterization via Coarse & Fine Weight Splitting

Low-precision CNN Model Quantization based on Optimal Scaling Factor Estimation

Neural Network Quantization: Separate Scaling of Rows and Columns in Weight Matrix

LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networks

Mixed-precision Deep Neural Network Quantization With Multiple Compression Rates

Divide and Conquer: Leveraging Intermediate Feature Representations for Quantized Training of Neural Networks

Training High-Performance and Large-Scale Deep Neural Networks with Full 8-Bit Integers.

Deep Neural Network Compression With Single and Multiple Level Quantization

Weighted-Entropy-Based Quantization for Deep Neural Networks