Abstract:Quantization Neural Networks (QNN) have attracted a lot of attention due to their high efficiency. To enhance the quantization accuracy, prior works mainly focus on designing advanced quantization algorithms but still fail to achieve satisfactory results under the extremely low-bit case. In this work, we take an architecture perspective to investigate the potential of high-performance QNN. Therefore, we propose to combine Network Architecture Search methods with quantization to enjoy the merits of the two sides. However, a naive combination inevitably faces unacceptable time consumption or unstable training problem. To alleviate these problems, we first propose the joint training of architecture and quantization with a shared step size to acquire a large number of quantized models. Then a bit-inheritance scheme is introduced to transfer the quantized models to the lower bit, which further reduces the time cost and meanwhile improves the quantization accuracy. Equipped with this overall framework, dubbed as Once Quantization-Aware Training~(OQAT), our searched model family, OQATNets, achieves a new state-of-the-art compared with various architectures under different bit-widths. In particular, OQAT-2bit-M achieves 61.6% ImageNet Top-1 accuracy, outperforming 2-bit counterpart MobileNetV3 by a large margin of 9% with 10% less computation cost. A series of quantization-friendly architectures are identified easily and extensive analysis can be made to summarize the interaction between quantization and neural architectures. Codes and models are released at <a class="link-external link-https" href="https://github.com/LaVieEnRoseSMZ/OQA" rel="external noopener nofollow">this https URL</a>

Two-Step Quantization for Low-bit Neural Networks

Propagating Asymptotic-Estimated Gradients for Low Bitwidth Quantized Neural Networks

Hessian-based Mixed-Precision Quantization with Transition Aware Training for Neural Networks

Quantization Networks

Bit-shrinking: Limiting Instantaneous Sharpness for Improving Post-training Quantization

Direct Quantization for Training Highly Accurate Low Bit-width Deep Neural Networks

Residual Quantization for Low Bit-Width Neural Networks.

Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networks

Stochastic Quantization for Learning Accurate Low-Bit Deep Neural Networks

LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networks

Towards Low-Bit Quantization of Deep Neural Networks with Limited Data.

Deep Neural Network Compression With Single and Multiple Level Quantization

A Novel Low-Bit Quantization Strategy for Compressing Deep Neural Networks

Bit Efficient Quantization for Deep Neural Networks

A White Paper on Neural Network Quantization

Learnable Companding Quantization for Accurate Low-bit Neural Networks

Low Precision Quantization-aware Training in Spiking Neural Networks with Differentiable Quantization Function

Once Quantization-Aware Training: High Performance Extremely Low-bit Architecture Search

SQuantizer: Simultaneous Learning for Both Sparse and Low-precision Neural Networks

Searching for Low-Bit Weights in Quantized Neural Networks

Training Multi-bit Quantized and Binarized Networks with A Learnable Symmetric Quantizer