Abstract:Recent advancements in machine learning achieved by Deep Neural Networks (DNNs) have been significant. While demonstrating high accuracy, DNNs are associated with a huge number of parameters and computations, which leads to high memory usage and energy consumption. As a result, deploying DNNs on devices with constrained hardware resources poses significant challenges. To overcome this, various compression techniques have been widely employed to optimize DNN accelerators. A promising approach is quantization, in which the full-precision values are stored in low bit-width precision. Quantization not only reduces memory requirements but also replaces high-cost operations with low-cost ones. DNN quantization offers flexibility and efficiency in hardware design, making it a widely adopted technique in various methods. Since quantization has been extensively utilized in previous works, there is a need for an integrated report that provides an understanding, analysis, and comparison of different quantization approaches. Consequently, we present a comprehensive survey of quantization concepts and methods, with a focus on image classification. We describe clustering-based quantization methods and explore the use of a scale factor parameter for approximating full-precision values. Moreover, we thoroughly review the training of a quantized DNN, including the use of a straight-through estimator and quantized regularization. We explain the replacement of floating-point operations with low-cost bitwise operations in a quantized DNN and the sensitivity of different layers in quantization. Furthermore, we highlight the evaluation metrics for quantized methods and important benchmarks in the image classification task. We also present the accuracy of the state-of-the-art methods on CIFAR-10 and ImageNet. This paper attempts to make the readers familiar with the basic and advanced concepts of quantization, introduce important works in DNN quantization, and highlight challenges for future research in this field.

Quantitative Effectiveness Assessment and Role Categorization of Individual Units in Convolutional Neural Networks

Quantitative Performance Assessment of CNN Units via Topological Entropy Calculation

Optimizing Quantized Neural Networks in a Weak Curvature Manifold

Multitask Convolutional Neural Network for No-Reference Image Quality Assessment

Analyzing and Quantifying Generalization in Convolutional Neural Networks

Sensitivity-Oriented Layer-Wise Acceleration and Compression for Convolutional Neural Network.

CNNComparator: Comparative Analytics of Convolutional Neural Networks.

Efficient Quantization for Neural Networks with Binary Weights and Low Bitwidth Activations.

Measuring Feature Importance of Convolutional Neural Networks.

Low-precision CNN Model Quantization based on Optimal Scaling Factor Estimation

Cnq: Compressor-Based Non-Uniform Quantization Of Deep Neural Networksinspec Keywordsother Keywordskey Words

Bag of Tricks with Quantized Convolutional Neural Networks for image classification

Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights

Role Taxonomy of Units in Deep Neural Networks

Visualizing and Comparing Convolutional Neural Networks

Learning Qualitative and Quantitative Image Quality Assessment.

Quantization Networks

Convolutional Neural Network with Nonlinear Competitive Units.

Quantization Effects on Neural Networks Perception: How would quantization change the perceptual field of vision models?

A Comprehensive Survey on Model Quantization for Deep Neural Networks in Image Classification

AutoQ: Automated Kernel-Wise Neural Network Quantization